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Foreword 


In  December  1998,  Navy  Personnel  Research  Studies  and  Technology  (NPRST) 
proposed  a  comprehensive  research  plan  to  better  attract,  employ,  and  retain  21st 
century  personnel.  Dubbed  Sailor  21,  the  program's  overarching  goal  was  to  develop  a 
series  of  models  and  tools  that  had  the  adaptive  capability,  predictability,  and  flexibility 
to  map  a  multivariate  predictor  space  (i.e.,  tests,  personality,  personal  history,  and 
interest)  onto  a  multivariate  performance  space  (NPRST,  1998).  This  agenda  proposed  a 
new  generation  of  selection  and  classification  processes  and  techniques  that  provide  a 
capability  to  evaluate  individual  sailor  traits  across  the  entire  personality  spectrum, 
ultimately  leading  to  a  definable  and  quantifiable  assessment  of  the  individual—  Whole 
Person  Assessment  (WPA).  To  develop  an  individual  vocational  interest  tool  as  part  of 
WPA,  researchers  at  NPRST  undertook  development  of  Job  Opportunities  in  the  Navy 
(JOIN)  (formerly  known  as  Jobs  and  Occupational  Interests  in  the  Navy).  This  work  was 
designed  to  investigate  potential  recruits’  vocational  interests  among  Navy  enlisted 
ratings.  This  report  marks  an  intermediate  step  in  producing  a  comprehensive  Navy  job 
(rating)  interest  tool,  one  that  affords  Navy  applicants  a  higher  degree  of  insight  into  the 
Navy's  world  of  work  and  offers  increased  understanding  of  Navy  jobs. 


David  L.  Alderton,  Ph.D. 
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Introduction 


Sailor  21 

In  December  1998,  Navy  Personnel  Research  Studies  and  Technology  (NPRST) 
proposed  a  comprehensive  research  plan  to  better  attract,  employ,  and  retain  21st 
century  personnel.  Dubbed  Sailor  21,  the  program's  overarching  goal  was  to  develop  a 
series  of  models  and  tools  that  had  the  adaptive  capability,  predictability,  and  flexibility 
to  map  a  multivariate  predictor  space  (i.e.,  tests,  personality,  personal  history,  and 
interest)  onto  a  multivariate  performance  space  (NPRST,  1998).  This  agenda  proposed  a 
new  generation  of  selection  and  classification  processes  and  techniques  that  provide  a 
capability  to  evaluate  individual  Sailor  traits  across  the  entire  personality  spectrum, 
ultimately  leading  to  a  definable  and  quantifiable  assessment  of  the  individual.  A  multi¬ 
dimensional  classification  technique,  Whole  Person  Assessment  (WPA)  includes 
capabilities  to  measure  potential  recruits’  (a)  testing  achievement,  (b)  complex 
reasoning,  (c)  spatial  ability,  (d)  job  specific  skills,  (e)  interests,  (f)  social  understanding, 
(g)  conscientiousness,  (h)  motivation,  (i)  leadership,  and  (j)  emotional  stability.  WPA 
and  its  subcomponents  optimally  match  prospective  Sailors  with  available  enlisted 
ratings  (jobs),  further  increasing  the  likelihood  of  a  more  congruent  fit  in  the  selection 
and  classification  process  (NPRST,  1998). 

To  develop  an  individual  vocational  interest  tool  as  part  of  WPA,  researchers  at 
NPRST  undertook  the  development  of  Job  Opportunities  in  the  Navy  (JOIN)  (formerly 
known  as  Jobs  and  Occupational  Interests  in  the  Navy).  This  work  was  designed  to 
investigate  potential  recruits’  vocational  interests  among  Navy  enlisted  ratings.  By  using 
personality  constructs  bound  up  in  person-environment  (P-E)  fit,  person-organization 
(P-O)  fit,  person-job  (P-J)  fit,  and  vocational  interest,  JOIN  potentially  increases 
Sailors’ job  satisfaction  (Klein  &  Wiener,  1977;  Worthington  &  Dolliver,  1977;  Rounds, 
Dawis,  &  Lofquist,  1987;  O’Reilly,  Chatman,  &  Caldwell,  1991;  Cronbach,  1984),  work 
performance  (Hogan  &  Blake,  1996),  organizational  commitment  (O’Reilly  et  al.,  1991; 
Chatman,  1991),  intrinsic  motivation  (Eisenberger,  Pierce,  &  Cameron,  1999;  Deci, 
Koestner,  &  Ryan,  1999),  leading  to  reduced  attrition  (Saks  &  Ashforth,  1997). 

Theories  of  Person-Environment  Fit 

The  purpose  of  implementing  any  vocational  interest  measure  within  an 
organization’s  recruitment,  selection,  and  classification  system  is  to  enhance  the  fit 
between  individuals  and  the  work  environment  (i.e.,  Person-Environment  fit;  also 
known  as  P-E  fit).  The  theory  of  P-E  fit  suggests  that  individual  attitudes  and  behaviors 
result  from  collaboration  between  persons  and  the  environment  (Edwards,  1996).  By 
enhancing  P-E  fit  between  individuals  and  work  environments,  benefits  such  as  job 
satisfaction  (Klein  &  Wiener,  1977;  Worthington  &  Dolliver,  1977;  Rounds  et  al.,  1987; 
O’Reilly  et  al.,  1991;  Cronbach,  1984),  organizational  commitment  (O’Reilly  et  al.,  1991; 
Chatman,  1991),  job  performance  (Hogan  &  Blake,  1996),  and  intrinsic  motivation 
(Eisenberger  et  al.,  1999;  Deci  et  al.,  1999)  may  be  derived.  Factors  such  as  turnover  are 
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reduced  by  enhancing  the  P-E  fit  between  an  individual  and  his/her  work  environment 
(Saks  &  Ashforth,  1997).  It  is  this  last  issue  of  excessive  turnover  (attrition)  that  is  of 
particular  concern  to  the  U.S.  Navy. 

There  are  a  number  of  different  types  of  P-E  Fit:  Person-Organization  Fit  (P-0  Fit) 
assesses  the  antecedents  and  consequences  of  compatibility  between  people  and  the 
organizations  in  which  (a)  at  least  one  of  the  entities  provides  what  the  other  needs,  (b) 
both  share  similar  fundamental  characteristics,  or  (c)  both  a  and  b  (Kristof,  1996). 
Person-Job  fit  (P-J  Fit)  emphasizes  the  compatibility,  contingency,  or  joint  influence  of 
the  person  and  job  for  predicting  individual  and  organizational  outcomes  (Edwards, 
1991).  It  focuses  on  the  correspondence  between  individual  desires  and  the  target  jobs. 

Throughout  the  academic  literature,  studies  have  demonstrated  the  benefits  of 
considering  P-J  fit;  research  has  shown  that  perceptions  of  P-J  fit  are  positively 
correlated  with  higher  job  satisfaction,  organizational  commitment,  and  organizational 
identification  and  negatively  correlated  with  stress  symptoms  and  intentions  to  quit 
(Saks  &  Ashforth,  1997).  Furthermore,  the  theory  of  decision  confirmation  suggests  that 
when  a  job  choice  experience  is  positive,  individuals  emphasize  information  that  leads 
to  positive  work  attitudes,  whereas  information  that  fosters  negative  work  attitudes  is 
discounted  or  ignored.  This  phenomenon  occurs  because  individuals  actively  seek  to 
confirm  positive  decisions  (Power  &  Aldag,  1985). 

One  of  the  most  commonly  studied  topics  in  this  arena  involves  the  impact  of  P-E  fit 
on  job  satisfaction.  Research  has  shown  that  high  vocational  interest  congruence  is 
highly  correlated  with  job  tenure  and  job  satisfaction  (Klein  &  Wiener,  1977). 
Worthington  and  Dolliver  (1977)  support  the  hypothesis  that  higher  congruence 
between  individuals’  interests  and  their  jobs  (person-job  congruence),  the  more  likely 
these  individuals  are  to  rate  themselves  as  satisfied  with  their  jobs.  Rounds  et  al.  (1987) 
showed  that  P-E  congruence  explains  3-30  percent  of  the  variance  in  job  satisfaction. 

Other  benefits  of  matching  individuals  with  organizations  (P-0  fit)  have  been 
supported  by  the  following  studies.  Saks  and  Ashforth  (1997)  suggest  that  perceptions  of 
P-0  fit  are  negatively  related  with  intentions  to  quit  and  turnover.  O’Reilly  et  al.  (1991) 
demonstrated  that  higher  P-0  fit  is  positively  correlated  with  organizational 
commitment  (/’  =  .25,  p  =  .01),  and  negatively  correlated  with  Intent  to  Leave  (r  =  -.37, 
p  =  .01).  Furthermore,  they  reported  that  the  measure  of  P-0  fit  is  positively  correlated 
with  job  satisfaction  (/’  =  .35,  p  <  .01). 

In  recent  literature  on  intrinsic  motivation,  General  Interest  Theory  (GIT)  proposes 
that  the  content  and  context  of  tasks  that  an  individual  performs  may  increase  intrinsic 
motivation  when  these  specific  tasks  help  satisfy  needs,  wants,  or  desires  (Eisenberger 
et  al.,  1999;  Deci  et  al.,  1999).  Therefore,  GIT  suggests  that  intrinsic  work  motivation 
may  be  enhanced  by  maximizing  the  fit  between  individuals  and  tasks  within  a  job.  In 
other  words,  enhancing  P-J  fit  may  generate  greater  intrinsic  motivation  among 
workers. 
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Vocational  Interests 

According  to  Savickas  and  Spokane  (1999),  E.  K.  Strong  documented  his  work  on  the 
measurement  of  vocational  interests  some  80  years  ago.  His  research  attempted  to 
answer  questions  such  as:  What  are  interests?  What  role  do  they  play  in  human  affairs? 
Can  one’s  behavior  be  predicted  if  the  individual’s  interests  are  known?  To  this  day, 
many  aspects  of  Strong’s  interest  theory  are  empirically  supported,  and  some  are 
hypotheses  waiting  to  be  investigated  by  future  researchers.  The  following  summarizes 
the  current  views  concerning  the  concept  of  vocational  interests. 

Perhaps  one  of  the  most  cited  definitions  of  vocational  interest  comes  from  Strong’s 
work  in  1943  in  which  he  defined  vocational  interests  as  job  related  activities  for  which 
we  have  liking  or  disliking,  are  attracted  to  or  repelled  by,  or  which  we  continue  or 
discontinue  from  status  quo;  furthermore,  they  may  or  may  not  be  preferred  to  other 
interests  and  may  continue  over  varying  intervals  of  time  (Strong,  1943).  Thus, 
vocational  interests  are  individual  preferences  (liking  or  disliking)  among  stimuli  (in 
this  case,  different  types  of  jobs)  that  lead  to  the  likelihood  of  an  individual  to  engage  in 
certain  activities.  In  other  words,  most  behaviors  are  the  result  of  individual  preferences 
for  engaging  in  specific  activities,  and  people  tend  to  go  toward  liked  activities  and  away 
from  disliked  activities  (Dawis,  1991).  Some  other  theorists  view  interests  as  the 
interaction  between  the  individuals  and  their  surrounding  environment.  In  this  sense, 
interests  are  the  collection  of  individuals’  experiences  in  terms  of  objects  and  activities 
that  yield  individual  tendencies  to  seek  personal  satisfactions  (Savickas,  1999). 

One  of  the  major  assumptions  underlying  the  relevant  research  literature  is  that 
interests  are  learned.  According  to  Savickas  (1999),  interests  derive  from  the  interaction 
between  individual  abilities  and  reinforcement  values— reinforcement  value  “denotes  a 
person’s  generalized  requirement  for  reinforcers  and  preference  for  stimulus  conditions 
that  in  the  past  have  been  reinforcing”  (Savickas,  1999).  On  the  other  hand,  abilities 
relate  to  the  individual’s  success  with  engaging  in  a  certain  activity  (Dawis,  1991).  Thus, 
positive  experiences  reinforce  interests  in  a  certain  area,  whereas  negative  experiences 
can  lead  to  the  rejection  of  that  particular  field.  In  summary,  the  origin  of  vocational 
interests  is  the  combination  of  the  individual’s  capacity  to  have  executed  activities  in  the 
past  and  the  value  of  the  current  reinforcer  (Savickas,  1999). 

Stability  of  Interests 

The  issue  of  stability  of  interests  was  succinctly  described  by  Fryer  (1931)  in  The 
Measurement  of  Interests  in  Relation  to  Human  Adjustment: 

To  hold  that  human  interests  are  stable,  that  interests  are  permanent, 
necessarily  denies  any  great  amount  of  variability  in  the  life  of  the 
individual.  But,  on  the  other  hand,  to  hold  that  interests  are  unstable, 
absolutely  lacking  in  permanence,  denies  any  possibility  of  the  genetic 
development  of  interests,  of  the  formation  of  habits  of  being  interested. 

(pp.  143-144) 
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Because  interests  are  the  synthesis  of  past  experiences  and  individual  abilities,  and 
both  of  these  components  are  subject  to  new  stimuli  that  generate  learning  behaviors, 
the  stability  of  interests  is  subject  to  change  over  time.  Some  research  shows,  however, 
that  interests  are  remarkably  stable  over  long  periods.  For  example,  Swanson  and 
Hansen  (1988)  investigated  the  longitudinal  stability  of  vocational  interests  in  a  sample 
of  409  college  freshmen  by  using  the  Strong-Campbell  Interest  Inventory  (SCII).  Their 
findings  suggest  that  interest  stability  remained  fairly  consistent  over  time,  with  mean 
stability  coefficients  of  .78,  .78,  and  .70  in  the  female  samples  and  .76,  .81,  and  .70  in  the 
male  samples  for  4-year,  8-year,  and  12-year  intervals,  respectively.  Cronbach  (1984) 
also  reported  that,  for  adolescents  who  are  not  bound  for  college,  interests  appear  to 
stabilize  comparatively  early.  Furthermore,  the  stability  of  individuals’  interests  after 
the  age  of  20  stayed  relatively  consistent  over  a  3-year  interval,  with  test-retest 
coefficients  of  0.75,  0.70,  and  0.70  for  participants  in  the  profession  of  biologist,  author- 
journalist,  and  office  worker,  respectively  (Cronbach,  1984).  Johansson  and  Campbell 
(1971)  summarized  Strong  Vocational  Interest  Blank  (SVIB)  data  and  found  that  test- 
retest  correlation  coefficients  ranged  from  0.54  to  0.84  over  the  intervals  of  1  to  23 
years. 

Despite  these  findings,  some  research  suggests  that  individual  interests  can  undergo 
substantial  change  over  time  (Swanson,  1999).  Furthermore,  it  seems  that  there  are  a 
number  of  variables  related  to  these  changes.  Swanson  (1999)  suggested  several 
hypotheses  regarding  factors  that  may  affect  interest  stability.  For  example,  stability 
appears  to  increase  with  age;  at  younger  ages,  interest  change  should  be  considered  as 
an  index  of  career  development  rather  than  instability.  Furthermore,  sex-typical 
interests  seem  to  be  more  stable  due  to  societal  support.  Thus,  stability  may  be  higher 
for  males  who  report  mechanical  or  scientific  interests,  and  for  females  who  report 
artistic  interests  (Stein,  Newcomb,  &  Bentler,  1986).  Next,  robust  interests  may  tend  to 
remain  more  stable  than  the  “lukewarm”  or  indifferent  interests.  Interest  instability  may 
also  be  associated  with  career  decisions  one  would  have  to  face,  where  a  change  in 
career  can  lead  to  the  willingness  to  consider  other  alternative  interest  areas  (Swanson, 
19991- 

In  summary,  future  studies  concerning  the  assessment  of  interest  should  address  the 
issue  of  stability.  For  example,  it  is  necessary  to  assess  the  stability  of  interest  to 
determine  whether  the  interests  of  Navy  Sailors  would  change  throughout  their  careers. 
Furthermore,  analyses  on  how  the  changes  of  interest  over  time  may  affect  Sailors’ 
organizational  commitment,  performance,  work  satisfaction,  turnover,  and  intrinsic 
motivation  should  also  be  included  in  future  studies  to  explore  the  utility  of  Job 
Opportunities  in  the  Navy  (JOIN)  beyond  the  first-term  of  enlistment.  Therefore,  future 
longitudinal  data  collection  efforts  will  be  crucial  for  the  long-term  assessment  of  JOIN. 

Overview  of  Navy  Recruiting,  Selection,  and  Classification  Procedures 

Each  year,  the  U.S.  Navy  must  select  and  classify  37,000-40,000  new  Sailors  to  fill 
its  manpower  needs  (Commander,  Navy  Recruiting  Command  statistics,  n.d.).  But 
today,  Navy  Recruiting  does  not  consider  any  sort  of  formal  vocational  interest  measure 
as  part  of  this  process.  What  information  is  known  is  obtained  informally  through  one- 
on-one  interviews  between  applicants,  enlisted  classifiers,  and  recruiters.  Of  principal 
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concern  here  is  to  what  extent  this  informal  approach  influences  downstream  retention. 
In  order  to  capture  the  nuances  of  this  sometimes  complicated  process,  a  brief  overview 
is  required. 

The  selection  and  classification  process  is  two-phased  and  two  Navy  actors  are  key  to 
leading  applicants  through  this  process— the  Navy  recruiter  and  the  enlisted  classifier. 
First  among  these  is  the  recruiter.  Recruiters  are  Sailors  assigned  to  duty  with  the 
Navy's  Recruiting  Command  who  make  initial  contact  with  prospective  applicants, 
establish  rapport,  conduct  preliminary  interviews,  and  perform  rudimentary  screening. 
During  initial  screening  recruiters  inquire  into  individuals'  backgrounds  on  topics  such 
as  citizenship,  education  attainment,  police  involvement,  medical  conditions,  etc. 
Additionally,  recruiters  administer  an  Enlistment  Screening  Test  (EST),  a  scaled  down 
version  of  the  Armed  Forces  Qualification  Test  (AFQT),  which  is  designed  to  assess 
verbal  and  mathematical  skills.  Administration  of  the  EST  provides  an  initial 
assessment  of  applicants'  intellectual  ability  and  it  functions  as  a  pre-screening  tool  to 
help  recruiters  determine  whether  individuals  meet  minimum  mental  ability 
requirements  for  entry.  Those  who  successfully  complete  the  EST  and  are  otherwise 
eligible  for  military  service  next  receive  full  cognitive  skill  testing  with  the  Armed 
Services  Vocational  Aptitude  Battery  (ASVAB).  The  AFQT  portion  of  the  test  uses  a 
series  of  cognitive  ability  tests  to  determine  selection  eligibility;  whereas,  ASVAB 
composites  are  used  principally  for  job  classification  purposes.  The  ASVAB  and  its 
AFQT  component  comprise  nine  distinct  subtests:  Word  Knowledge  (WK),  Paragraph 
Comprehension  (PC),  Arithmetic  Reasoning  (AR),  Mathematics  Knowledge  (MK), 
General  Science  (GS),  Auto  and  Shop  Information  (AS),  Mechanical  Comprehension 
(MC),  Electronics  Information  (El)  and  Assembling  Objects  (AO).  Upon  completion  of 
basic  eligibility  screening  and  ASVAB  testing,  applicants  proceed  to  the  second  phase  of 
processing. 

Phase  two  of  the  recruitment  process  takes  place  at  a  Department  of  Defense  (DoD) 
sponsored  and  U.S.  Army  operated  activity— the  Military  Entrance  Processing  Station 
(MEPS).  At  MEPS,  candidates  for  enlistment  are  given  a  thorough  physical  examination, 
undergo  in-depth  background  investigation,  and  may  receive  specialized  testing.  Those 
who  successfully  complete  all  of  the  Navy's  entry  requirements  are  then  given  the 
opportunity  to  meet  and  discuss  job  options  with  a  job  counselor  or  enlisted  classifier  in 
Navy  vernacular.  During  this  brief  counseling  session,  a  list  of  available  jobs  as 
determined  by  ASVAB  scores  and  job  vacancies  is  presented  to  the  candidate.  Potential 
recruits  are  provided  with  a  short  one-page  description  of  the  job,  the  type  of  work,  and 
a  brief  description  of  where  that  work  may  be  performed.  The  applicant  is  given  an 
opportunity  to  ask  questions  of  the  classifier  about  the  work  involved  and  discuss 
alternatives,  bonus  options,  term  of  enlistment,  etc.  At  the  end  of  this  session, 
individuals  are  asked  to  make  a  selection.  It  is  during  this  process  that  enlisted 
classifiers,  who  operate  under  an  assigned  recruiting  goal,  often  guide  applicants 
towards  vacancies  that  fit  Navy  needs,  but  give  little  consideration  to  applicant's 
preferences. 

If  the  applicant  successfully  navigates  all  of  the  aforementioned  obstacles,  a  contract 
is  drafted  with  the  specification  of  an  individual  work  assignment  (rating),  term  of 
enlistment,  and  any  applicable  bonuses  that  may  be  offered.  At  this  point  candidates 
proceed  to  a  swearing-in  ceremony  and  then  typically  return  home  to  await  their  date  of 
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departure  (i.e.,  ship  date).  On  rare  occasions,  individuals  will  proceed  directly  from 
swearing-in  to  awaiting  transportation  for  the  journey  to  Recruit  Training  Command 
Great  Lakes,  Illinois  to  begin  basic  training. 

Problem  of  Attrition 

During  the  years  1985-1995  Navy  first-term  attrition  (defined  as  the  proportion  of 
enlistees  who  fail  to  complete  their  first  contractual  obligation)  increased  steadily  from 
30.6  percent  to  39.6  percent  (GAO,  2000).  Approximately  13.6  percent  of  all  enlistees 
were  separated  before  they  completed  6  months  of  service.  Of  those  individuals,  roughly 
39.1  percent  of  all  attrites  in  1995  exited  the  Navy  within  6  months  of  their  initial 
enlistment.  Furthermore,  the  estimated  average  cost  of  recruiting  and  training  each 
enlistee  rose  from  $19,143  in  1993  to  $35,500  in  1998.  Therefore,  if  the  recent  attrition 
rate  persists,  the  cost  to  recruit  and  train  new  Navy  Sailors  will  become  even  greater 
(GAO,  2000).  Currently,  the  Navy  lacks  data  concerning  the  causes  for  first-term 
attrition,  but  one  possibility  in  light  of  other  research  is  that  it  is  caused  by 
inappropriate  job  classifications. 

Development  of  a  Navy  Vocational  Interest  Inventory 

In  an  effort  to  improve  the  classification  process  and  reduce  attrition,  the  Navy 
undertook  the  development  of  a  vocational  interest  inventory  known  as  Job 
Opportunities  in  the  Navy  (JOIN).  Designed  to  supplement  information  obtained  from 
the  ASVAB  and  recruiter  interviews,  JOIN  measures  vocational  interests  of  Navy 
recruits  vis-a-vis  the  various  enlisted  ratings.  Derived  from  the  theories  of  vocational 
interest,  P-E,  P-O,  and  P-J  fit,  it  is  a  tool  that  directly  addresses  the  increasing  problem 
of  attrition.  JOIN  is  designed  to  be  used  in  concert  with  the  Rating  Identification  Engine 
(RIDE),  an  algorithm  that  uses  individuals'  ASVAB  scores  to  generate  a  list  of  ratings  for 
which  individuals  are  qualified  (the  instrument  does  not  display  ratings  that  are  not 
vacant).  In  other  words,  RIDE  has  the  capability  to  receive  inputs  from  JOIN  and  rank 
order  the  qualified  ratings  according  to  individuals’  self-reported  interests  (dependent 
upon  the  availability  of  each  rating).  Overall,  RIDE  (with  the  capability  to  input  rating 
preference  data  from  JOIN)  will  be  used  by  the  Navy  classifiers  to  distribute  manpower 
among  the  wide  array  of  Navy  work  (Farmer  et  al.,  2007). 

Development  of  the  First  JOIN  Prototype  (JOIN  1.0  Delta) 

JOIN  is  a  computerized  interest  inventory  designed  to  assess  recruits’  vocational 
interests  with  respect  to  four  domains  of  the  Navy  jobs  (community  area,  work  style, 
work  environment,  and  work  activity;  see  Appendix  A).  The  instrument  is  structured  to 
be  a  flexible  inventory  that  evaluates  recruits’  preferences  with  respect  to  these  domains. 
The  initial  idea  was  to  use  existing  vocational  interest  inventories  developed  by  the 
Armed  Forces,  private  organizations,  and  other  government  agencies  as  resources  for 
the  JOIN  item  development.  Instruments  such  as  Vocational  Interest  Career 
Examination  (VOICE),  Army  Vocational  Interest  Career  Examination  (AVOICE),  Navy 
Vocational  Interest  Inventory  (NVII),  Vocational  Preference  Inventory  (VPI),  and  many 
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others  were  used  to  help  guide  the  development  of  JOIN  (Farmer  et  al.,  2007).  By  using 
the  hierarchical  model  of  work  development  (Schippmann,  1999),  analyses  among  job 
families  (e.g.,  aviation,  construction,  intelligence,  etc.),  work  styles  (e.g.,  mental, 
physical,  etc.),  work  environments  (e.g.,  indoor,  outdoor,  physical,  mental,  etc.),  and 
work  activities  (analyze  communications,  direct  aircraft,  maintain  documents,  etc.)  were 
conducted  to  guide  the  development  of  the  survey  items.  Next,  a  project  plan  was 
created  to  list  the  steps  for  item  development  and  assessment  of  validity  and  reliability. 
The  overall  concept  was  to  construct  an  instrument  that  would  discriminate  across  all 
entry-level  ratings  to  increase  P-J  fit.  Furthermore,  JOIN  is  intended  to  serve  as  a  tool  to 
inform  potential  recruits  of  the  Navy’s  world-of-work  (hence  P-0  and  P-E  fit). 
Administration  time  and  readability  of  items  were  also  major  concerns  throughout  the 
development  of  the  instrument,  due  to  the  limited  amount  of  time  available  at  MEPS  for 
JOIN  administration  and  the  diverse  education  level  of  recruits. 

Due  to  the  diverse  education  level  of  recruits  (ranging  from  GED  to  college 
graduates),  descriptions  of  items  were  written  in  a  simple  and  clear  manner.  In  addition, 
pictures  were  collected  to  complement  each  instrument  item.  Furthermore,  because 
most  recruits  have  little  prior  knowledge  about  the  range  of  tasks  among  Navy  ratings,  a 
broad  spectrum  of  each  job  activity  was  depicted  by  a  brief  behavioral  descriptive  with 
pictures  to  ensure  that  all  aspects  of  job  activities  were  included.  Because  each 
participant  receives  two  types  of  stimuli  (description  and  pictures),  the  inclusion  of  both 
pictures  and  textual  description  helps  to  minimize  biased  responses  due  to  the 
misrepresentation  of  items.  In  other  words,  the  provision  of  pictures  along  with 
behavioral  descriptions  of  each  item  assists  the  potential  recruits  with  understanding 
the  JOIN  items. 

Item  Development 

The  first  step  in  the  item  development  was  to  compile  all  available  job  descriptions 
for  the  79  entry-level  enlisted  ratings  from  sources  such  as  the  U.S.  Navy  homepage, 
Navy  Manpower  and  Personnel  homepage,  Job  Information  Cards,  and  Enlisted 
Community  Managers  (ECMs).  These  descriptions  were  extracted  to  create  preliminary 
survey  items  that  represent  each  entry-level  rating  among  the  four  different  domains: 
community  area,  work  style,  work  environment,  and  work  activity.  With  regard  to  work 
activity,  facets  within  the  domain  were  divided  into  two  separate  components:  (1)  work 
process  (e.g.,  to  analyze,  maintain,  operate,  etc.)  and  (2)  work  content  (documents, 
mechanical  equipments,  weapons,  etc.).  The  primary  purpose  of  separating  the  facets 
within  the  work  activity  domain  into  two  components  was  to  create  a  logical  structure 
among  the  overall  job  activities  within  the  Navy,  as  well  as  to  produce  a  smaller  number 
of  work  activity  items.  Furthermore,  the  revision  of  survey  items  can  be  more  effortless 
because  the  work  activity  items  (i.e.,  Process-Content  pairs;  P-C  pairs)  can  be  easily 
added  or  deleted.  The  main  objective  of  the  P-C  pairs  is  to  discriminate  aspects  of  the 
Navy  enlisted  job  activities  from  one  another  (Watson,  Michael,  Hindelang,  Farmer,  & 
Alderton,  2006;  see  Appendix  A  of  this  report  for  a  list  of  the  P-C  pairs). 

Next,  multiple  feedback  sessions  were  conducted  with  ECMs  to  review  and  revise  the 
items.  A  number  of  ECMs  within  each  community  area  participated  in  this  Subject 
Matter  Expert  (SME)  validation  exercise.  To  ensure  the  resemblance  of  inventory  items 
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to  the  realistic  characteristics  of  each  job,  items  were  revised  according  to  the 
suggestions  provided  by  the  SMEs.  The  task  was  to  ask  each  ECM  to  indicate  activities 
that  were  considered  important  to  any  specified  ratings.  For  example,  analyses  showed 
that  the  essential  work  activities  of  an  Electrician’s  Mate  (EM)  include  analyze- 
documents,  maintain-electrical  equipment,  maintain-mechanical  equipment,  operate- 
electrical  equipment,  and  operate-mechanical  equipment.  The  overall  objective  of  this 
phase  of  research  was  to  generate  face  validity  of  the  instrument  and  further  enable  the 
individual  to  discriminate  interests  among  different  ratings.  In  other  words,  the  JOIN 
l.oie  instrument  items  were  postulated  to  resemble  existing  entry-level  ratings.  In  this 
version  of  JOIN,  the  instrument  consists  of  7  community  areas,  4  work  styles,  4  work 
environments,  and  26  work  activities  (See  Appendix  A  for  the  list  of  JOIN  facets  within 
each  domain). 

Pictorial  Item  Development 

After  initial  item  formation,  the  next  phase  of  instrument  development  consisted  of 
the  identification  and  selection  of  photographs  that  represent  each  of  the  survey  items 
provided  by  numerous  departments  of  the  Navy  (e.g.,  Navy  Recruiting  Command)  and 
their  Internet  sources.  These  pictures  serve  as  supplements  to  the  descriptions  for  each 
work  dimension.  The  general  premise  was  to  develop  an  interest  inventory  that  goes 
beyond  the  textual  descriptions  of  traditional  interest  inventories.  Based  on  the 
assumption  that  most  potential  recruits  have  little  knowledge  of  Navy  entry-level 
ratings,  these  pictures  represent  realistic  images  of  the  Navy’s  community  areas,  work 
styles,  work  environments,  and  work  activities.  In  addition,  they  also  assist  in  the 
understanding  of  each  item’s  textual  description  and  hence  minimize  any  biased 
responses  related  to  partial  exposure  of  the  Navy’s  world-of-work  from  the  media  or 
other  non-related  sources.  Because  this  type  of  exposure  can  provide  only  a  portion  of 
the  Navy’s  world-of-work,  three  criteria  were  determined  to  guide  the  selection  of 
images.  These  criteria  specified  that  each  set  of  pictures  would  need  to  (1)  be 
representative  of  both  routine  and  non-routine  job  tasks;  (2)  incorporate  job-specific 
activities  performed  in  all  aspects  of  community  area,  work  style,  work  environment, 
and  work  activities;  and  (3)  display  both  gender  and  racial  diversity  (Watson  et  al., 
2006).  Approximately  500  pictures  were  collected  to  represent  all  facets  within  the  job 
domains.  Subsequently,  these  images  were  assessed  by  two  groups  of  SMEs  to  validate 
the  visual  content  of  the  collected  images.  The  overall  objective  was  to  ask  experts  to 
categorize  each  picture  in  accordance  to  the  predetermined  domains  and  facets  (see 
Appendix  A),  and  also  identify  any  dimension  that  was  overlooked  from  previous  item 
development.  This  validation  effort  resulted  in  more  than  300  images  within  JOIN 
l.oie,  with  three  to  four  pictures  representing  each  item  (See  Appendix  B  for  examples 
of  instrument  items). 

Software  Development 

Programmers  affiliated  with  Electronic  Data  Systems  (EDS)  developed  the  JOIN 
software  using  Visual  Basic.  The  effort  resulted  in  the  formation  of  the  first  prototype  of 
the  Navy  computer-administered  interest  inventory,  JOIN  1.0  Delta.  The  inventory 
consists  of  three  components. 
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The  first  portion  of  the  inventory  consists  of  an  item  that  asks  respondents  to  rank 
order  preferences  among  community  areas  (e.g.,  aviation,  construction,  submarine,  etc). 
Second,  questions  were  presented  regarding  each  of  the  elements  within  work  styles 
(e.g.,  work  with  teams,  work  independently,  etc.)  and  work  environments  (e.g.,  indoor, 
outdoor,  etc.).  More  specifically,  participants  were  asked  to  rate  each  item  on  a  graphic 
rating  scale  from  o  to  100  (higher  numbers  correspond  with  higher  the  interest).  Third, 
three  sets  of  work  activity  items  (i.e.,  the  work  activity,  Maintain-Weapons,  has  3  items 
consisting  of  identical  behavioral  descriptions  but  different  pictures)  were  administered 
to  respondents  randomly  in  accordance  with  the  26  work  activities  (e.g.,  Maintain- 
Documents,  Operate-Weapons,  etc.).  The  scale  for  these  items  was  also  from  o  to  100. 

Initial  Validation  of  the  First  JOIN  Prototype  (JOIN  1.0  Delta) 

The  first  JOIN  prototype  (JOIN  1.0  Delta)  was  administered  to  300  new  Navy 
recruits  during  their  third  or  fifth  day  after  arriving  at  basic  training.  During  this  time, 
they  were  gathered  by  divisions  (approximately  80-100  recruits)  in  an  open-bay  area 
for  medical  and  dental  in-processing  (including  dental  and  visual  examinations, 
immunization,  and  sex  education).  Those  individuals  who  were  waiting  to  be  called  for 
processing  or  immediately  following  processing  were  asked  to  participate  in  pilot 
testing.  This  particular  sample  was  hypothesized  to  have  the  closest  match  to  future 
users  of  the  instrument  because  the  majority  of  recruits  do  not  possess  a  great  deal  of 
knowledge  regarding  the  world-of-work  within  the  Navy.  However,  some  response  bias 
may  have  occurred  because  these  recruits  have  already  selected  a  rating.  In  other  words, 
the  knowledge  acquired  from  recruitment  may  have  generated  response  bias,  even 
though  this  exposure  may  be  minimal. 

Results  of  the  initial  pilot  testing  suggest  that  the  first  prototype  of  JOIN  (JOIN  1.0 
Delta)  is  construct  valid  and  internally  reliable  (Watson  et  al.,  2006).  In  reference  to  the 
original  model  design  of  JOIN,  the  items  should  be  able  to  discriminate  among  the 
different  Navy  enlisted  ratings.  To  test  this  assertion,  a  principal  component  analysis 
was  conducted  on  the  work  activities  to  ensure  the  items  can  distinguish  and 
discriminate  across  the  Navy  enlisted  ratings  (construct  validity).  According  to  Watson 
et  al.  (2006),  the  analysis  produced  a  9-component  solution  that  accounted  for  89 
percent  of  the  variance  among  the  Navy  enlisted  ratings  (e.g.,  electronic,  mechanical, 
administrative,  mass  communication,  construction,  intelligence,  and  weapons 
activities).  In  addition,  the  resultant  factors  seemed  to  correspond  to  the  previously 
established  occupational  groups. 
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Descriptive  analyses  were  performed  to  establish  the  overall  distribution  of 
participants’  responses  on  each  of  the  items.  For  community  areas,  approximately  33 
percent  of  the  respondents  expressed  the  highest  interest  in  the  aviation  community. 
About  18  percent  of  the  respondents  selected  special  programs  to  be  their  top  choice. 
Health  care  and  intelligence  were  the  next  highest  responses,  with  14  percent  of 
respondents  ranking  these  dimensions  to  be  their  preferred  community. 

Work  style  and  work  environment  items  were  assessed  in  the  same  fashion.  For  work 
styles,  most  participants  indicated  a  preference  for  working  with  a  team  and  working 
with  physical  tasks.  For  work  environments,  outdoor  activities  showed  the  highest 
interest  among  respondents  in  comparison  to  indoor,  office,  and  industrial 
environments. 

Analyses  concerning  work  activity  items  were  also  conducted.  As  mentioned  above, 
three  sets  of  individual  items  were  developed  for  each  work  activity.  Internal  consistency 
of  the  item  sets  was  assessed  to  ensure  the  items  were  measuring  a  single  dimension. 
Analyses  concerning  the  alpha  reliability  of  the  work  activity  items  showed  an  average 
internal  consistency  of  0.895  among  all  work  activity  items.  Moreover,  the  individual 
dimensions  among  each  work  activity  indicated  alpha  levels  ranging  from  0.830 
(Operate  Mechanical  Equipment)  to  0.945  (Make  Facilities),  with  the  median  between 
0.898  (Maintain  Security)  and  0.905  (Train  People). 

During  administration  of  the  initial  pilot  test,  a  Usability  Feedback  Survey  was  given 
to  assess  different  aspects  of  the  first  JOIN  prototype.  These  exploratory  Likert-scaled 
items  consisted  of:  ease  of  use,  clarity  of  instructions,  readability  of  description,  ratings 
of  the  tutorial,  visual  appeal,  etc.,  which  were  designed  to  evaluate  the  face  validity  of 
JOIN  (Watson  et  al.,  2006).  Descriptive  analyses  suggest  that  JOIN  1.0  Delta  is  easy  to 
use  and  visually  appealing.  The  instructions  and  tutorial  ratings  tended  to  be  either 
good  or  very  good.  And  most  participants  responded  very  positively  when  comparing 
JOIN  to  traditional  paper-and-pencil  questionnaires.  However,  a  majority  of 
participants  indicated  that  JOIN  1.0  Delta  does  not  hold  users’  attention  very  well. 
Feedback  from  open-ended  questions  suggests  that  the  work  activity  items  seem 
redundant  due  to  the  similarities  among  some  of  the  work  activity  dimensions.  For 
example,  descriptions  and  images  for  items  such  as:  maintain  mechanical  equipment, 
make  mechanical  equipment,  and  operate  mechanical  equipment  were  very  similar  to 
each  other.  Some  participants  were  unable  to  make  any  distinctions  among  these  items. 
Concerns  have  also  been  addressed  from  Navy  classifiers  to  keep  the  administration 
time  to  approximately  15  minutes.  Minimizing  administration  time  is  crucial  to  ensure 
that  recruitment  procedures  at  MEPS  are  not  lengthened  drastically  from  the  current  3- 
4  hour  process.  Indications  from  this  negative  feedback  suggest  that  a  shorter  version  of 
JOIN  might  be  more  appropriate. 

To  address  the  problem  of  redundancy  and  administration  time,  a  subsequent 
version  was  developed  (JOIN  l.oie),  which  consists  of  two  sets  of  work  activity  items 
(item-couplets)  rather  than  three  (item-triplets).  There  is  a  need  to  determine  the 
psychometric  properties  of  this  new  iteration.  Specifically,  comparisons  among  internal 
consistency  of  the  work  activity  items  between  the  JOIN  1.0  Delta  (item-triplets)  and 
JOIN  l.oie  (item-couplets)  need  to  be  conducted  to  assess  whether  the  reduction  of 
items  lowers  the  internal  consistency  of  JOIN  significantly. 
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Discussion  groups  were  conducted  to  generate  feedback  from  participants.  In  all,  11 
discussion  groups  (57  recruits)  were  conducted  throughout  the  3-day  data  collection 
process.  The  results  were  very  similar  to  responses  from  the  Usability  Feedback  Survey; 
repetitive  items  should  be  eliminated  to  increase  the  ability  of  JOIN  to  hold  participants’ 
attention  throughout  the  administration  process. 

Instrument  Algorithm 

After  respondents  answer  all  questions  in  JOIN,  scores  are  automatically  calculated 
using  an  internal  algorithm.  The  algorithm  of  JOIN  l.oie  uses  test  results  and  SME 
determined  weightings  in  the  calculation  of  individual  interest  fit  scores  (JOIN  Fit)  for 
each  Navy  enlisted  rating.  Beginning  with  the  sum  of  squares  among  the  products  of 
participants’  responses  and  the  facet  scoring  weights  within  each  domain  ("Facet 
Scores"),  computation  of  Euclidean  distances  among  each  “Facet  Scores”  result  in  a 
series  of  “Preference  Data  Scores”  among  the  79  Navy  enlisted  entry-level  ratings.  These 
“Preference  Data  Scores”  are  defined  as:  (1)  functional  category  (i.e.,  community  area), 
(2)  environment  preference  (i.e.,  work  style  and  work  environment),  and  (3)  process 
preference  (i.e.,  work  activity).  Next,  the  “Preference  Data  Scores”  are  combined  to 
produce  a  “JobRankValue.”  The  smaller  the  “JobRankValue,”  the  higher  the  Person-Job 
congruence.  Each  “JobRankValue”  is  converted  into  a  JOIN  Fit  score.  This  procedure 
results  in  the  establishment  of  JOIN  Fit  scores,  for  which  the  higher  the  value  of  JOIN 
Fit,  the  greater  the  individual  interest  fit  to  that  specific  rating.  In  other  words,  JOIN  Fit 
represents  the  degree  of  congruence  between  interests  of  individuals  and  the  79  Navy 
enlisted  ratings;  each  individual  receives  79  JOIN  Fit  scores  that  represent  the  fit  of 
Sailors  to  each  rating.  (See  Appendix  C  for  a  detailed  description  of  the  JOIN  Job 
Ranking  Formula  and  Appendix  D  for  an  example  calculation.) 


Purpose 


This  study  consists  of  three  objectives:  the  first  is  to  reevaluate  the  internal 
consistency  among  the  work  activity  items  (also  known  as  Process-Content  Pairs  [P-C 
pairs]  or  item-couplets)  to  assess  whether  the  reliabilities  of  the  P-C  items  are  sufficient 
after  item  reduction.  More  specifically,  analyses  of  Cronbach’s  Alpha  are  conducted  to 
assess  whether  the  internal  consistency  of  the  work  activity  items  are  adequate  to  be 
applied  to  the  Navy’s  workforce.  Moreover,  to  ensure  that  JOIN  does  not  produce 
different  results  among  subgroups  of  individuals,  comparisons  of  internal  consistency 
among  gender  and  race  (BUPERS,  n.d.)  are  investigated  to  determine  whether  JOIN 
l.oie  is  more  reliable  for  any  specific  subgroup. 

Hypothesis  1:  A  reduction  in  the  number  of  work  activity  items  results  in  an  internal 
consistency  among  the  items  of  at  least  a  =  .70. 

At  the  time  of  data  analysis,  all  participants  had  completed  at  least  basic  training. 
Some  portion  of  the  recruits  may  have  also  completed  their  job  training  (Apprentice 
School  Training).  Therefore,  the  second  purpose  of  this  study  was  to  concurrently 
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validate  JOIN  l.oie  using  data  from  the  Navy  First  Watch  program,  which  assesses  self- 
reported  training  success  and  satisfaction  of  recruits  at  each  of  the  initial  milestones 
(i.e.,  basic  training  arrival,  graduation,  apprentice  school  graduation,  and  exiting  the 
Navy  force).  The  current  study  hypothesizes  that  JOIN  l.oie  is  significantly  correlated 
with  training  success  and  satisfaction— that  recruits  are  more  satisfied  and  more  likely  to 
succeed  in  training  when  the  fit  between  the  individual  and  his  or  her  rating  is  high. 

Hypothesis  2:  JOIN  Fit  scores  generated  from  JOIN  l.oie  correlate  positively  and 
significantly  with  training  success  and  satisfaction. 

Exploratory  analyses  are  conducted  to  identify  the  changes  for  the  next  version  of 
JOIN.  Specifically,  a  modified  version  of  the  Usability  Feedback  Survey  is  administered 
throughout  data  collection  to  gather  feedback  from  recruits.  The  expectation  is  that  the 
reduction  of  work  activity  items  (item-  triplet  approach  reduced  to  the  item-couplet 
approach)  lessens  the  proportion  of  participants  who  perceive  the  items  as  repetitive. 

Hypothesis  3:  Lower  number  of  participants  perceive  the  work  activity  items  of  JOIN 
l.oie  as  repetitive ,  in  comparison  to  the  findings  from  JOIN  1.0  Delta. 


Method 


Participants 

Based  on  previous  power  analyses  of  Chi-square  and  multiple  regression  criterion 
validation  methodologies,  the  calculation  suggested  that  a  large  sample  of  3,000-5,000 
subjects  was  necessary  to  identify  clear  predictors  for  the  instrument.  As  a  result,  JOIN 
l.oie  was  administered  to  approximately  4,500  recruits  during  their  first  few  days  of 
basic  training  at  Recruit  Training  Center  (RTC),  Great  Lakes,  IL.  Random  sampling  was 
not  feasible  for  this  data  collection  effort  due  to  the  way  in  which  recruits  are  processed. 
Therefore,  a  convenience  sample  is  used  throughout  this  study.  Participants’  social 
security  numbers  (SSNs)  were  collected  in  order  to  track  these  individuals  for  future 
longitudinal  studies  in  criterion  performance,  once  the  data  become  available  and 
analyzable. 

Instrumentation 

JOIN  l.oie  was  modified  based  on  feedback  collected  from  the  JOIN  1.0  Delta 
Usability  Feedback  Survey.  The  JOIN  l.oie  model  is  the  same  instrument  as  JOIN  1.0 
Delta  except  for  a  reduction  in  the  number  of  work  activity  items.  This  alteration  is 
postulated  to  reduce  the  likelihood  that  participants  perceived  the  instrument  as 
repetitive.  The  content  of  JOIN  l.oie  consists  of:  1  multiple  response  community  item,  8 
work  environment  items,  and  2  sets  of  26  work  activity  items  (item-couplets).  All 
responses  collected  from  participants  are  stored  electronically  in  an  Microsoft  Excel 
spreadsheet.  Data  have  been  converted  into  SPSS  format  for  statistical  analyses. 
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Procedure 


Prior  to  administering  JOIN  l.oie,  a  pre-inventory  script  was  read  to  participants 
(See  Appendix  E  for  the  pre-inventory  script).  Materials  from  the  pre-inventory  script 
consisted  of  a  short  description  of  JOIN  l.oie  along  with  the  basic  nature  of  questions 
appearing  in  the  instrument.  Statements  about  confidentiality  and  voluntary 
participation  were  provided  to  meet  current  ethical  standards  of  research. 

Throughout  this  study,  JOIN  l.oie  was  administered  to  recruits  on  the  third  or  fifth 
day  after  arriving  at  RTC.  Similar  to  previous  validation  efforts,  Navy  recruits  were 
gathered  by  divisions  (approximately  80-100  future  Sailors)  in  an  open-bay  area  and 
were  called  for  medical  and  dental  in-processing  that  included  eye  examination, 
vaccination,  health  education,  dental  check-up,  and  dental  hygiene.  Those  individuals 
who  were  waiting  to  be  called  for  processing  or  immediately  following  processing  were 
asked  to  participate  in  this  study.  JOIN  l.oie  took  approximately  15  minutes  to 
complete. 

The  first  1,000  participants  received  a  Usability  Feedback  Survey  that  assessed 
different  aspects  of  JOIN  l.oie.  Specifically,  the  combination  of  Likert-scaled  items  and 
open-ended  questions  evaluated  the  ease  of  use,  clarity  of  instructions,  readability  of 
descriptions,  ratings  of  the  tutorial,  and  visual  appeal  of  the  instrument.  These  items 
served  as  the  basis  for  future  modifications.  (See  Appendix  F  for  a  copy  of  the  Usability 
Feedback  Survey) 

For  the  convenience-sampled  discussion  groups,  roughly  six  to  eight  participants 
were  asked  to  provide  feedback  regarding  their  understanding  of  instructions,  images, 
job  descriptions,  and  area  differentiation  (i.e.,  community  area  versus  work-styles 
versus  work  activities,  etc.).  These  discussion  groups  were  held  immediately  following 
JOIN  l.oie  and  Usability  Feedback  Survey  administrations.  Information  collected  from 
these  groups  was  utilized  to  determine  the  programs’  functionality  under  certain 
organizational  constraints  (e.g.,  available  test  time,  recruit  characteristics).  The 
Usability  Feedback  Survey  followed  by  a  group  discussion  required  an  additional  10-15 
minutes  to  complete. 

Instrument  Reliability 

The  reliability  portion  of  the  study  contained  two  parts.  First,  reassessments  of 
internal  consistencies  among  the  work  activity  items  were  required  to  examine  and  to 
compare  the  alpha  level  of  the  current  work  activity  couplets  to  the  original  work  activity 
triplets.  Due  to  reductions  in  process-content  (P-C)  items,  it  was  necessary  to  assess  the 
reliability  coefficients  to  ensure  that  the  reduction  in  work  activity  items  did  not  impose 
any  detrimental  effect  on  the  internal  consistency  of  the  instrument.  Alpha  reliability 
analyses  were  conducted  on  each  of  the  26  P-C  pairs  (item-couplets)  to  determine 
whether  the  reduction  of  items  would  sustain  a  good  or  reasonable  level  of  internal 
consistency. 

Second,  subgroup  differences  on  internal  consistency  were  also  examined. 

Evaluation  of  subgroup  differences  among  genders  and  racial  groups  were  assessed  to 
ensure  that  the  instrument  was  no  less  reliable  or  valid  for  any  specific  subgroup.  By 
using  the  following  formula  provided  in  Bonett  (2003),  the  alpha  coefficients  between 
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the  gender  and  racial  subgroups  were  compared  to  determine  whether  the  internal 
consistencies  among  subgroups  were  statistically  significant. 

T_ _ In (S) _ 

^  r  ~\V 

2  kx  2k 2  72 

_  (K  ~  l)(n2  -  2)  +  {k2  - 1)(«2  -  2)  _ 

where  5  =  — — — 

(1-  P2) 

According  to  Bonett  (2003),  the  above  equation  may  be  used  to  compare  “two 
independent  groups  of  size  m  and  n2  where  pi  is  (the  internal  consistency)  estimated  in 
Group  1  from  a  test  having  ki  parts  and  p2  is  estimated  in  Group  2  from  a  test  having  k2 
parts.”  (p.  73)  For  statistical  analysis,  the  researcher  evaluated  z-scores  generated  from 
the  equation  with  za/2  =  1.96  (za/2  represents  a  2-tailed  critical  z-value  for  the  specified  a 
level).  More  precisely,  if  z  >  za/2,  then  pi  <  p2;  if  z  <  -za/2,  then  pi  >  p2;  otherwise,  the 
results  were  statistically  insignificant. 

Instrument  Validity 

The  assessment  of  the  validity  of  JOIN  was  a  two  part  process.  First,  predictive 
validation  analyses  were  conducted  to  assess  the  potential  impact  of  the  instrument. 
Correlation  analyses  were  used  to  survey  the  relationship  between  individual  job  fit  (in 
accordance  with  their  rating)  and  NPRST  First  Watch  data  (including  Exit  Survey,  New 
Sailor  Survey,  RTC  Graduate  Survey,  and  Apprentice  School  Graduation  Survey) 
collected  during  recruits’  basic  and  job  training.  More  specifically,  five  items  from  the 
New  Sailor  Survey,  four  items  from  the  RTC  Graduate  Survey,  four  items  from  the 
Apprentice  School  Graduation  Survey,  and  two  items  from  the  Exit  Survey  were  used  for 
the  predictive  validation  effort  (see  Appendix  G  for  the  First  Watch  items).  These  items 
covered  topics  such  as  expectation  of  training  success,  satisfaction  with  the  classification 
process,  job  satisfaction,  and  motivation  to  leave  the  Navy. 

Second,  a  modified  version  of  the  Usability  Feedback  Survey  was  provided  to  the  first 
1,000  research  participants  to  gather  feedback  concerning  numerous  facets  of  JOIN 
l.oie.  Similar  to  the  previous  survey,  items  such  as  ease  of  use,  instruction  rating, 
information,  and  content  rating  were  given  in  Likert-type  scaling.  Other  questions  such 
as  the  amount  of  weekly  computer  usage  prior  to  arriving  at  basic  training  and  open- 
ended  questions  were  also  asked  to  determine  whether  further  revision  of  JOIN  was 
warranted.  Items  concerning  special  skills  (e.g.,  musical  ability,  foreign  language  and 
special  license  or  certification)  were  included  to  assess  the  need  to  include  items  that 
correspond  with  special  skills  ratings  (e.g.,  Musicians).  By  conducting  analyses  of 
variance  (ANOVA)  and  correlation  analyses  using  the  Usability  Feedback  Survey,  this 
segment  of  instrument  validation  could  serve  as  an  assessment  of  the  face  validity  of 
JOIN. 
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Results 


Internal  Consistency  (Alpha  Reliability)  Analyses 

To  determine  the  acceptability  of  work  activity  item  reduction  from  three  pairs  to 
two  pairs,  a  reassessment  of  the  internal  consistencies  was  conducted.  Cronbach’s  alpha 
was  calculated  on  each  individual  pair  of  work  activity  items  (item-couplets).  Results 
showed  that  the  average  internal  consistency  among  all  26  item-couplets  was  .854  (N  = 
4,348;  Appendix  H),  which  was  a  minor  decrease  from  .895  of  the  original  prototype 
with  item-triplets.  In  addition,  the  previous  reliability  estimates  for  individual  work 
activity  items  ranged  between  .830  (Operate  Mechanical  Equipments)  to  .945  (Make 
Facilities).  The  current  version  of  item-couplets  estimated  a  range  between  .768  (Make 
Communications)  to  .914  (make  facilities).  With  300  subjects  randomly  selected  from 
the  total  4,348  cases,  alpha  estimates  ranged  between  .776  (Make  Communications)  to 
.924  (Maintain  Security).  In  fact,  this  simple  numeric  comparison  has  shown  that  the 
item  reduction  effort  increased  the  alpha  reliability  of  two  work  activities  (Maintain 
Security  and  Maintain  Weapons)  when  comparing  the  original  287  participants  from  the 
previous  version  to  the  current  version  with  4,348  cases  (See  Appendix  I).  With  the 
randomly  selected  300  cases  (300  of  4,348),  the  alpha  levels  for  Maintain  Security, 
Maintain  Weapons,  Make  Documents,  and  Operate  Weapons  increased.  The  change  in 
internal  consistency  for  individual  work  activities  ranged  from  -.050  to  .091  when 
comparing  the  original  287  cases  to  the  current  4,348  cases.  For  the  randomly  selected 
300  cases,  changes  in  the  alpha  levels  ranged  between  -.070  to  .086.  With  the  exception 
of  Direct  Emergency  Response,  Make  Facilities,  Make  Communications,  and  Operate 
Electronic  Equipment,  the  changes  for  work  activity  items  did  not  exceed  .07  when 
compared  to  the  previous  version  (N  =  287).  The  changes  in  alpha  when  comparing 
nearly  equal  cases  (287  vs.  300)  did  not  exceed  0.07,  with  the  exception  of  Maintain 
Facilities,  Make  Communication,  Respond  To  Emergencies,  and  Serve  Customers. 

Subgroup  Comparisons  on  Alpha  Reliability 

After  analyses  of  internal  consistencies  among  all  identifiable  minority  and  gender 
groups,  results  suggested  that  the  instrument  maintained  a  high  level  of  internal 
consistency  among  subgroups.  For  example,  the  self-identified  Asian-American, 
African-American,  Hispanic-American,  and  Caucasian-American  participants  averaged 
.847,  .842,  .849,  and  .856  on  all  26  process-content  item  pairs  (item-couplets), 
respectively  (Appendix  J).  Among  all  items  from  the  Asian-American  subgroup,  Operate 
Mechanical  Equipment  showed  the  lowest  internal  consistency  of  .706  while  Make 
Facilities  showed  the  highest  at  .922.  Within  the  African-American  subgroup,  Operate 
Electronic  Equipment  (a  =  .771)  showed  the  lowest  internal  consistency  and  Operate 
Weapons  (a  =  .904)  the  highest.  For  Hispanic-American  participants,  the  least  reliable 
item-couplet  was  Make  Communications  with  a  =  .744,  and  the  most  reliable  was 
Operate  Weapons  with  a  =  .929.  Lastly,  Caucasian-American  participants  received 
higher  average  internal  consistency  for  the  work  activity  items  (a  =  .856)  than  other 
subgroups.  The  range  of  internal  consistency  for  white  participants  was  between  .767 
(Make  Communications)  and  .927  (Make  Facilities).  In  order  to  discover  whether  any 
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differences  among  racial  subgroups  were  significant,  statistical  comparisons  among 
independent  alpha  coefficients  (Bonett,  2003)  were  conducted.  Results  showed  that 
there  were  no  statistical  differences  among  racial  subgroups  concerning  the  overall 
average  alpha  coefficient  of  work  activity  items  within  JOIN  l.oie  (Appendix  K).  In 
other  words,  the  statistical  results  suggested  that  the  differences  among  racial  groups 
were  statistically  insignificant. 

Comparison  of  internal  consistencies  between  the  two  gender  groups  presented 
positive  results.  With  averaged  reliability  coefficients  of  .847  and  .854  for  respective 
male  and  female  subgroups,  the  least  reliable  item-couplet  for  male  participants  was 
Make  Communications  (.754),  and  Maintain  Facilities  (.772)  for  females  (Appendix  L). 
The  most  consistent  item-couplet  was  Make  Facilities  (a  =  .910)  for  the  male 
participants,  and  Operate  Office  Equipment  (a  =  .924)  for  females.  Statistical 
comparison  on  the  two  alpha  coefficients  suggested  that  the  difference  between  the  two 
gender  groups  was  not  statistically  significant  (z  =  .494;  Appendix  K). 

Predictive  Validity  of  JOIN  l.Ole 

As  a  result  of  characteristic  anomalies  in  the  First  Watch  surveys  (lower  value 
represents  higher  traits),  reverse  coding  was  required  for  all  analyzed  data.  Correlation 
analyses  among  JOIN  Fit  scores  and  cross-sectional  survey  data  were  conducted.  None 
of  the  Exit  Survey  items  was  significantly  associated  with  JOIN  Fit  scores.  Throughout 
the  New  Sailor  Survey,  the  expectation  of  recruits’  success  in  training  was  positively 
correlated  with  JOIN  Fit  scores  (r  =  .051,  p  =  .032;  Appendix  M).  Sailor  responses 
indicated  that  satisfaction  with  training  school  guaranteed  at  classification  (prior  to 
entry)  was  positively  correlated  with  JOIN  Fit  score  (r  =  .059,  p  =  .012).  Contrary  to 
expectations,  greater  preparation  for  basic  training  was  negatively  associated  with  JOIN 
Fit  (r  =  -.049,  p  =  .040).  Furthermore,  no  significant  correlations  were  found  between 
JOIN  Fit  and  satisfaction  with  the  amount  of  time  spent  with  a  job  counselor  (i.e., 
enlisted  classifier)  (r  =  -.034,  p  =  .150)  and  the  amount  of  information  provided  on  the 
assigned  classification  (r  =  .042,  p  =  .076). 

Within  the  RTC  Graduate  Survey,  there  were  four  items  significantly  associated  with 
JOIN  Fit.  The  extent  to  which  the  following  items  were  explained  to  the  recruits  resulted 
in  positive,  significant  correlations:  Navy  jobs  available  to  you  at  graduation 
(r  =  0.065,  P  =  0.021),  job  that  was  assigned  at  classification  (r  =  .059,  p  =  .036),  and 
school  that  was  guaranteed  at  classification  (r  =  .076,  p  =  .007).  In  contrast,  individuals 
who  were  reclassified  or  wished  to  be  reclassified  during  boot  camp  training  were 
significantly  and  negatively  correlated  with  JOIN  Fit  scores  (r  =  -.179,  p  <  .001).  Lastly, 
results  from  the  Apprentice  School  Graduation  Survey  indicated  that  greater  satisfaction 
with  their  current  rating  was  significantly  related  with  higher  JOIN  Fit  scores  (r  =  .149, 
p  =  .009). 
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Usability  Feedback 

A  usability  feedback  survey  was  designed  and  employed  to  offer  insight  into  the 
practical  usefulness  of  JOIN  l.oie.  Descriptive  analyses  on  the  Likert-scaled  items 
indicated  that  93.7  percent  and  90.9  percent  of  the  participants  evaluated  Ease  of  Use 
and  Instructions  as  good  or  very  good,  respectively.  Based  on  similar  aggregation  of 
good  or  very  good  ratings,  the  majority  of  participants  also  rated  the  following  items 
positively:  tutorials  (80.9%),  visual  appeal  (85.9%),  ability  to  hold  the  attention  of 
participants  (66%),  understanding  the  pictorial  items  representation  (93.6%),  and 
perceived  increase  of  knowledge  due  to  the  usage  of  the  instrument  (70.3%).  Next, 
almost  all  participants  indicated  that  JOIN  l.oie  was  equal  to  or  better  than  paper- 
pencil  format  inventories  (96.9%).  For  face  validity,  91.3  percent  felt  that  the  pictures 
related  well  or  very  well  with  the  descriptions  provided  in  JOIN  l.oie.  Furthermore, 

84.7  percent  of  participants  indicated  that  the  combination  of  pictures  and  descriptions 
is  more  informative  than  text  alone.  The  adjusted  mean  (after  removing  participants 
outside  of  3  standard  deviations)  for  average  time  spent  on  computers  prior  to  basic 
training  was  17.19  hours,  with  88.3  percent  of  participants  rating  themselves  as  average 
to  expert  users. 

Inferential  statistical  tests  were  conducted  to  assess  significant  relationships  among 
the  survey  items.  Analyses  of  variance  (ANOVA)  and  post  hoc  analyses  (Tukey  HSD  & 
Bonferroni)  indicated  that  there  were  significant  differences  between  the  individuals 
who  used  both  pictures  and  text  to  rate  the  quality  of  instructions.  Participants  who 
used  both  text  and  pictures  to  determine  their  responses  during  the  administration  of 
JOIN  l.oie  rated  the  instrument’s  instructions  higher  than  those  using  text  or  pictures 
alone  (F( 2,  961)  =  9.143,  p  <  .001;  Appendix  N).  Tukey  HSD  and  Bonferroni  post  hoc 
analyses  suggested  that  participants  rated  the  tutorials  of  JOIN  l.oie  significantly 
higher  when  using  both  text  and  pictures  to  respond  to  JOIN  items  than  when  using 
textual  information  alone  (F( 2,  959)  =  5.087,  p  =  .006).  Furthermore,  Tukey  HSD, 
Bonferroni,  and  Games-Howell  post  hoc  analyses  agreed  that  JOIN  l.oie  held 
participants’  attention  better  when  the  participants  used  both  pictures  and  text  to 
evaluate  their  responses  than  when  using  the  textual  descriptions  only  (F( 2,  959)  = 
5.390,  p  =  .005).  Participants  responded  more  positively  on  their  understanding  of  the 
information  presented  in  each  picture  when  they  used  both  text  and  picture  rather  than 
these  components  individually  (F( 2,  974)  =  9.365,  p  <  .001;  post  hoc  agreement  among 
Tukey  HSD,  Bonferroni,  &  Games-Howell).  Significant  differences  were  found  on  the 
perceived  amount  of  knowledge  gained  by  using  JOIN  l.oie  when  individual  responses 
were  based  on  both  textual  (descriptions)  and  pictorial  (images)  stimuli,  or  based  on 
judgments  of  these  stimuli  independently,  (F( 2,  970)  =  13.294,  p  <  .001).  Post  hoc 
analyses  using  Tukey  HSD,  Bonferroni,  and  Games-Howell  agreed  that  subjects  who 
evaluated  their  responses  using  both  textual  descriptions  and  pictorial  images  rated 
significantly  higher  on  perceived  knowledge  (among  Navy  enlisted  ratings)  gained  than 
subjects  who  used  text  alone  (p  <  .001).  Lastly,  results  suggested  that  the  individuals 
who  based  their  responses  on  the  pictorial  stimuli  alone  perceived  they  had  gained  more 
knowledge  after  the  administration  of  JOIN  l.oie  than  individuals  who  based  their 
responses  on  textual  information  alone  (p  =  .005). 
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In  summary,  when  individuals  view  textual  and  pictorial  stimuli,  they  tended  to 
perceive  JOIN  l.oie  as  (l)  having  high  quality  instructions  and  tutorials,  (2)  better  able 
to  hold  participants’  focus  throughout  survey  administration,  and  (3)  more  capable  of 
inducing  high  levels  of  understanding  about  the  jobs  within  the  Navy’s  enlisted 
community  (in  comparison  to  traditional  paper-pencil  vocational  interest  inventories). 
The  statistical  outcomes  suggest  that  the  images  in  JOIN  provide  significant 
understanding  about  Navy  ratings  compared  with  text-only  job  descriptions.  These 
results  indicate  that  JOIN  1.0  le  has  a  high  degree  of  face  validity. 

Non-partial  correlation  analyses  were  conducted  among  the  usability  feedback  items 
to  discover  any  indication  of  how  participants  responded  to  the  JOIN  items.  Results 
show  that  responses  between  individual  rating  on  the  Ease  of  Use  item  and  its  visual 
appeal  suggested  that  these  two  items  correlated  at  the  .337  level  (p  <  .001).  In  addition, 
the  Ease  of  Use  rating  was  highly  correlated  with  the  individual  ratings  of  instruction  (r 
=  -634,  p  <  .001)  and  tutorials  (r  =  .451,  p  <  .001).  Participants  tended  to  rate  the 
instructions  more  positively  when  the  tutorial  ratings  were  higher  (r  =  .537, 
p  <  .001).  Time  spent  on  using  computers  was  positively  correlated  with  the  self- 
assessment  of  individual  computer  skill  level  (r  =  .389,  p  <  .001). 

Because  participants  reported  that  items  from  JOIN  1.0  Delta  were  repetitive,  a 
regression  analysis  was  conducted  to  explore  variables  that  affected  the  amount  of 
attention  JOIN  l.oie  was  holding  among  individuals.  Stepwise  linear  regression 
(Appendix  O)  indicates  that  the  visual  appeal  of  the  instrument  accounts  for  31  percent 
of  the  variance  on  how  JOIN  l.oie  holds  individuals'  attention  during  administration 
CF(i,  845)  =  370.177,  p  <  .001).  The  second  step  consists  of  the  Visual  Appeal  item 
rating  and  the  item  “How  did  JOIN  do  at  increasing  your  knowledge  of  Navy  work  areas, 
styles,  and  activities?”  Results  indicate  that  the  combination  of  these  variables 
accounted  for  40  percent  of  variance  on  how  JOIN  l.oie  holds  the  attention  of 
respondents  (F( 2,  844)  =  269.952,  p  <  .001).  The  third  step  included  an  additional 
variable  Instructions  rating  that  accounts  for  40.3  percent  of  the  variance  on  how  JOIN 
l.oie  holds  the  attention  of  participants  (F( 3,  843)  =  189.403,  p  <  .001).  Lastly,  the 
combination  of  four  items:  Visual  Appeal  rating,  “How  did  JOIN  do  at  increasing  your 
knowledge  of  Navy  work  areas,  styles,  and  activities,”  Instruction  rating,  and  “how  does 
JOIN  compare  to  other  paper-pencil  formatted  instruments”  accounts  for  41  percent  of 
the  total  variance  on  JOIN  1. oie's  ability  to  hold  participants  attention  (F( 4,  842)  = 
146.451,  p  <  .001).  The  stepwise  analysis  excluded  several  moderately  to  highly 
correlated  variables  such  as  Ease  of  Use  rating  (r  =  .267,  p  <  .001),  Tutorial  rating  (r  = 
•358 ,p  <  .001),  “how  the  description  relates  to  the  pictures”  (r  =  .311,  p  <  .001),  and 
“how  well  could  you  understand  what  was  being  presented  in  each  picture”  (r  =  .327,  p  < 
.001)  because  these  variables  do  not  provide  any  incremental  variance  to  the  criterion 
(i.e.,  ability  to  hold  participants  attention).  If  all  variables  mentioned  are  included 
within  the  regression  analysis,  these  items  would  have  accounted  for  41.7  percent  of  the 
total  variance  on  the  amount  of  attention  JOIN  l.oie  holds  from  participants  (F(  10, 

836)  =  59.870,  p  <  .001).  In  summary,  the  overall  results  of  the  stepwise  linear 
regression  analysis  showed  that  the  amount  of  attention  JOIN  l.oie  holds  for 
participants  was  highly  dependent  upon  the  visual  appeal  of  the  instrument,  the  amount 
of  knowledge  JOIN  l.oie  provides,  and  the  perceived  quality  of  the  instructions. 
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Furthermore,  when  individuals  rated  JOIN  l.oie  as  a  better  assessment  tool  than  other 
traditional  paper-pencil  inventories,  it  also  added  some  incremental  variance  to  the 
ability  of  JOIN  l.oie  to  hold  participants’  attention  throughout  administration. 


Discussion 


It  is  hypothesized  that  the  Navy  selection  and  classification  system  generates  a 
proportion  of  Sailors  who  are  misfit  to  their  assigned  ratings/jobs.  As  a  result, 
development  of  an  interest  inventory  that  creates  higher  person-organization  and 
person-job  fit  is  necessary  to  fill  this  gap  in  the  Navy’s  selection  and  classification 
system.  In  an  attempt  to  reduce  attrition  by  increasing  work  satisfaction,  organizational 
commitment,  work  performance,  and  intrinsic  motivation,  JOIN  was  hypothesized  to 
reduce  a  major  part  of  selection,  classification,  training,  and  attrition  costs  by 
maximizing  the  match  between  individual  interests  and  assigned  Navy  ratings. 
Therefore,  the  overall  purpose  of  JOIN  is  to  assess  the  work  interests  of  recruits  who 
have  volunteered  to  enter  the  Navy’s  world-of-work,  as  well  as  to  place  these  recruits 
into  the  ratings  where  they  are  more  likely  to  be  satisfied,  committed,  motivated,  and 
perform  at  a  higher  level. 

Results  from  this  phase  of  instrument  validation  produced  several  promising 
outcomes.  First,  the  internal  consistency  of  the  instrument,  after  the  reduction  of  work 
activity  items,  remained  strong.  As  mentioned  previously,  the  main  difference  between 
the  first  JOIN  prototype  (JOIN  1.0  Delta)  and  a  subsequent  version  (JOIN  l.oie)  is  the 
number  of  process-content  items  within  each  work  activity:  the  current  version  was 
reduced  from  item-triplets  for  each  work  activity  to  item-couplets.  Comparisons 
between  these  two  variants  of  JOIN  indicate  that  the  work  activity  item-couplets 
continued  to  produce  medium  to  high  internal  consistency  after  the  item  reduction. 

With  an  overall  average  of  .854  on  alpha  reliability  among  all  work  activity  items,  the 
instrument  maintained  a  good  level  of  internal  consistency.  Even  the  most  inconsistent 
item  (make  communications)  produced  an  alpha  reliability  of  .768  within  the  item- 
couplets.  Therefore,  current  data  suggest  that  work  activity  items  of  JOIN  l.oie  produce 
high  levels  of  reliability  among  participants.  With  limited  time  available  to  administer 
JOIN  (Navy  classifiers  suggest  keeping  administration  time  less  than  20  minutes),  JOIN 
l.oie  achieves  more  than  acceptable  levels  of  internal  consistencies  with  the  average 
administration  time  between  15-20  minutes. 

Stemming  from  a  large  number  of  participants  (N  =  4,348)  in  the  JOIN  l.oie 
dataset,  assessment  of  internal  consistency  among  the  work  activity  item-couplets  with 
a  sample  of  300  randomly  selected  participants  was  necessary  to  compare  these  data 
with  the  previous  287  sampled  participants  from  JOIN  1.0  Delta.  The  reduction  in 
sample  size  did  not  detriment  the  internal  consistency  of  the  instrument  greatly.  Results 
showed  that  with  300  randomly  selected  cases  (from  a  pool  of  4,384  participants),  the 
average  overall  alpha  was  .856,  with  the  lowest  a  =  .776  (Make  Communications)  and 
highest  a  =  .924  (Maintain  Security).  The  current  reliability  estimates  resulted  in  a 
medium  to  high  level  of  internal  consistency.  Moreover,  the  instrument  did  not  show 
any  significant  differences  on  the  internal  consistencies  among  individual  subgroups 
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(i.e.,  racial  groups  and  gender).  In  other  words,  although  minor  numerical  differences 
were  present  among  the  racial  and  gender  subgroups,  there  were  no  statistically 
significant  differences.  More  importantly,  these  analyses  may  be  used  as  baseline  data 
for  future  studies  on  whether  JOIN  will  produce  adverse  impact,  even  though  the 
instrument  is  designed  to  supplement  rather  than  replace  the  current  selection  and 
classification  system. 

Predictive  validity  of  JOIN  l.oie  shows  optimistic  results.  Correlation  analyses 
indicate  that  individual  JOIN  Fit  scores  generated  from  the  algorithm  of  the  instrument 
were  positively  correlated  with  self  expectation  of  basic  training  success,  satisfaction 
with  guaranteed  rating  (job),  amount  of  job-related  information  provided  at 
classification,  satisfaction  with  the  assigned  rating,  satisfaction  with  the  “A”  school 
guaranteed  at  classification,  and  satisfaction  with  the  individual  rating  at  the  time  of 
administration.  Furthermore,  JOIN  Fit  was  negatively  correlated  with  individuals  who 
have  been  or  have  requested  to  be  reclassified  during  basic  training.  Although  the 
relationships  are  small,  the  results  suggest  that  the  instrument  may  become  a  valuable 
supplement  to  the  current  selection  and  classification  system;  JOIN  l.oie  adds 
additional  information  concerning  how  satisfied  the  individuals  are  with  their  classified 
ratings,  self-expectation  of  training  success  and  job  satisfaction.  Using  JOIN  as  a 
supplement  to  the  current  evaluation  of  individuals’  cognitive  ability  and  other 
psychological  and  physiological  factors  will  enable  the  Navy  to  incorporate  non- 
cognitive  assessments  (specifically,  vocational  interests  assessment)  into  the  process  of 
selection  and  classification.  Consequently,  when  new  recruits  are  classified  into  ratings 
that  have  high  vocational  interest  fit,  they  will  more  likely  be  satisfied  with  their  job, 
more  committed  to  the  organization,  and  perform  better,  as  well  as  being  more 
intrinsically  motivated  to  their  jobs.  In  addition,  the  Navy’s  attrition  rate  is  also  likely  to 
decline  (Saks  &  Ashforth,  1997). 

Once  all  previous  participants  have  completed  Class  A  technical  school  training  and 
their  initial  enlistment  period  (as  specified  on  their  contract),  an  evaluation  of  attrition 
and  satisfaction  among  this  cohort  may  be  possible.  When  these  data  become  available, 
predictive  and  concurrent  validation  can  continue  to  assess  long-term  effects  of  JOIN  on 
the  Navy  selection  and  classification  system. 

The  assessment  of  usability  of  JOIN  l.oie  provides  positive  feedback  concerning 
many  aspects  of  the  instrument.  A  majority  of  the  participants  indicated  that  JOIN  l.oie 
was  easy  to  use,  with  good  instructions  and  tutorials  to  assist  in  understanding  the 
activities  involved  in  administration.  Furthermore,  respondents  indicated  that  the 
instrument  contains  good  visual  appeal  and  holds  participants'  attention.  The  pictures 
were  understandable  and  relevant  to  the  descriptions  provided.  Roughly  70  percent  of 
the  participants  expressed  that  they  gained  significant  knowledge  about  Navy  ratings 
after  taking  JOIN  l.oie.  Collectively,  these  descriptive  outcomes  indicate  that  JOIN 
l.oie  may  be  a  useful  tool  that  provides  knowledge  concerning  the  Navy  enlisted  ratings, 
and  assesses  vocational  interests  among  these  ratings. 

Analysis  of  variance  (ANOVA)  results  from  the  usability  survey  indicate  that  the 
images  included  within  JOIN  served  as  a  better  tool  than  the  descriptive  text  alone. 
Moreover,  participants  responded  more  positively,  gained  more  knowledge,  and  paid 
more  attention  when  the  images  were  used  along  with  the  descriptions  to  determine 
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their  responses.  Correlation  analyses  confirmed  some  intuitive  relationships.  For 
example,  JOIN’S  ease  of  use  rating  was  highly  correlated  with  individual  ratings  of 
instructions  and  tutorials.  Furthermore,  JOIN  Fit  scores  were  moderately  correlated 
with  the  visual  appeal  of  the  instrument.  Amount  of  time  spent  using  computers  was 
moderately  correlated  with  individual  self-assessment  of  computer  skill  level.  Even 
though  most  of  these  relationships  were  intuitive,  they  provided  indications  that  the 
instructions,  tutorials,  and  images  affect  how  people  perceive  the  ease  of  use  of  JOIN 
items. 

Stepwise  linear  regression  analysis  demonstrated  that  the  amount  of  attention  JOIN 
l.oie  held  from  participants  was  largely  dependent  upon  (l)  the  visual  appeal  of  the 
instrument,  (2)  the  amount  of  perceived  knowledge  concerning  the  Navy’s  world-of- 
work,  (3)  ratings  of  instructions,  and  (4)  whether  the  participants  viewed  JOIN  as  better 
than  other  paper-pencil  formatted  instruments.  Therefore,  in  order  to  minimize  the 
amount  of  attention  lost  from  the  perceived  repetitive  items,  continuous  improvement 
on  the  visual  items  should  be  pursued.  Future  researchers  should  persist  with  assessing 
other  methods  of  providing  instructions  that  may  result  in  more  positive  feedback. 

Although  statistical  outcomes  were  positive  with  respect  to  the  internal 
consistencies,  face  validity,  and  predictive  validity  of  JOIN  l.oie,  four  types  of  future 
studies  may  assist  in  validating  this  interest  inventory.  First,  even  though  correlation 
analyses  were  conducted  to  discover  the  relationships  between  JOIN  Fit  scores  and 
other  self-reported  performance  and  satisfaction  items  from  First  Watch,  actual 
performance  and  satisfaction  scores  were  not  readily  analyzable  at  this  point.  Therefore, 
it  will  be  necessary  to  investigate  the  relationship  between  JOIN  Fit  scores  and  actual 
records  of  training  and  work  performance  (Hogan  &  Blake,  1996),  training  and  job 
satisfaction  (Klein  &  Wiener,  1977;  Worthington  &  Dolliver,  1977;  Rounds,  Dawis,  & 
Lofquist,  1987;  O’Reilly,  Chatman,  &  Caldwell,  1991;  Cronbach,  1984),  organizational 
commitment  (O’Reilly  et  al.,  1991;  Chatman,  1991),  and  intrinsic  motivation 
(Eisenberger,  Pierce,  &  Cameron,  1999;  Deci,  Koestner,  &  Ryan,  1999).  These  studies 
will  assist  in  discovering  the  true  benefits  of  generating  Fit  by  using  JOIN  (Edwards, 
1991).  Secondly,  because  of  substantial  first-term  enlisted  attrition,  a  utility  analysis  is 
warranted  to  determine  the  return  on  investment  (ROI)  of  developing  JOIN  to  assist  in 
classification  of  Navy  enlisted  recruits.  More  specifically,  an  evaluation  of  the  amount  of 
expenditures  saved  by  using  JOIN  to  reduce  first-term  attrition  may  provide  further 
support  for  the  value  of  JOIN  as  a  supplement  to  Navy  selection  and  classification 
system.  Third,  future  studies  are  warranted  to  investigate  specific  items  that  seemed 
repetitive.  Evaluation  of  the  responses  to  these  seemingly  repetitive  items  may  allow 
researchers  to  understand  how  participants  answer  these  specific  items  when  perceived 
as  repetitive.  Lastly,  evaluation  on  the  usefulness  of  interest  inventories  developed  using 
a  job  analysis  method  outside  a  Navy  enlisted  population  will  assist  in  generalizing  the 
approach  used  in  JOIN.  An  extension  of  JOIN  should  include  additional  components 
that  relate  to  other  military  services  and  civilian  jobs,  and  further  include  the  capability 
to  assess  and  rank  interests  among  all  available  work  within  the  military  and  civil 
service  systems. 


21 


Concerning  the  implementation  of  JOIN,  existing  literature  suggests  that  the 
interpretation  of  results  can  be  key  to  the  overall  success  of  JOIN.  For  example, 

Zytowski  (1999,  p.  277-293)  indicated  that  when  recruiters  interpret  vocational  interest 
inventory  results  for  job  applicants,  five  principal  concepts  should  be  used:  (1)  prepare 
for  the  discussion  of  results  by  familiarizing  themselves  with  the  large  and  small  details 
of  the  inventory,  (2)  involve  prospective  recruits  in  communicating  the  results,  (3)  use 
simple,  but  emphatic,  language  or  illustrations  throughout  discussion,  (4)  ask 
prospective  recruits  to  recapitulate  their  results  by  using  their  own  words,  and  (5) 
stimulate  continuing  career  development  by  identifying  steps  or  methods  for  exploring 
career  options  suggested  throughout  the  discussion  of  the  assessment  results.  According 
to  Goodyear  (1990),  appropriate  interpretation  of  vocational  interest  assessment  results 
may  stimulate  client  satisfaction,  as  well  as  increase  exploratory  behavior  and  enhance 
vocational  identity  and  career  maturity.  Put  differently,  by  using  these  principles,  the 
Navy  can  ensure  that  the  utility  of  JOIN  is  maximized. 
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Appendix  A: 
JOIN  Facets 


A-o 


JOIN  Facets 


Navy  Community  Areas 

Work  Activities 

Aviation 

Analyze  Communications 

Construction 

Analyze  Data 

Health  Care 

Analyze  Documents 

Intelligence 

Direct  Aircraft 

Submarine 

Direct  Emergency  Response 

Surface 

Maintain  Documents 

Special  Programs 

Maintain  Electrical  Equipment 

Maintain  Electronic  Equipment 

Maintain  Facilities 

Maintain  Mechanical  Equipment 

Maintain  Security 

Maintain  Supplies 

Maintain  Weapons 

Make  Communications 

Work  Styles/Work  Environments 

Make  Documents 

Mental 

Make  Facilities 

Physical 

Make  Mechanical  Equipment 

Independent 

Operate  Electrical  Equipment 

Team 

Operate  Electronic  Equipment 

Indoors 

Operate  Facilities 

Outdoors 

Operate  Mechanical  Equipment 

Industrial 

Operate  Office  Equipment 

Office 

Operate  Weapons 

Respond  to  Emergencies 

Serve  Customers 

Train  People 
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Appendix  B: 

Example  Screenshots  of  JOIN  l.Ole 
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[jobs  and  Occupational  Interest  in  the  Navy  (JOIN) _ 

Community  Preference 


[jobs  and  Occupational  Interest  in  the  Navy  (JOIN) 


Next  >> 


Ask  for  help 


Less  Interesting 

Next  >> 


Navy  jobs  are  organized  in  the  Community  areas  shown  below.  Please  click  on  each  community  you  are 
interested  in  and  move  each  to  the  right  with  the  arrow  button.  Then  use  the  arrows  on  the  right  to  rank  order 
your  communities  of  interest. 


Not  Interesting 


More  Interesting 


Special  Programs 
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[jobs  and  Occupational  Interest  in  the  Navy  (JOIN) 


Work  Style  Preference 


Very  Interested 


Ask  for  help 


Next  >> 
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Appendix  C: 

JOIN  Job  Ranking  Algorithm 
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JOIN  Job  Ranking  Formula 


There  are  three  sets  of  recruit  preference  data 

(i)  Functional  Category  (OSG  or  community) 

(ii)  Process  Preference 

(iii)  Environment  Preference 

These  three  categories  are  assumed  to  represent  fundamentally  different  aspects  of  a 
job,  and  therefore  each  is  used  to  derive  a  single  value  (weight)  before  the  three  values 
are  combined  to  give  an  overall  job  ranking. 

For  category  (i)  -  Functional  Category,  retrieve  the  ranking  in  the  preference  set  (set 
arbitrary  large  value  (e.g.,  99)  for  those  functional  areas  not  preferred).  The  weight  Wk 
for  job  i  is  calculated  as 

Wii  =  min(rii)  for  OSGs  preferred  by  candidate,  where  ru  are  the  preference 
ordinals  for  all  the  OSGs/communitiesj  in  which  job  i  is  a  member 
(and  rj  =  99  for  OSGs/communities  which  are  not  preferred) 

For  category  (ii)  -  Process  Preference,  where  candidate  preference  is  scored  0-11,  it 
is  assumed  that  the  set  of  variables  represents  a  spanning  set  across  the  job  space,  and 
that  each  variable  is  independent  and  equally  weighted.  Then  derive  a  combined  value 
for  Process  Space  Preference  as  the  normed  ’’distance”  between  the  candidate 
preferences  and  the  job  attributes  [referred  to  as  Deoxyribonucleic  acid  (DNA)  in  this 
case,  an  underlying  pattern  structure  of  variables  o  or  1].  This  value  (W2i  for  job  i)  will 
be  between  o  and  1. 

W2i  =  [2P(cp  -  jip)2]1/2  /  P1/2 

where  the  sum  is  over  the  P  process  attributes  (variables) 
cp  is  the  candidate  preference  value  for  attribute  p 
jiP  is  the  job  DNA  value  for  job  i  for  attribute  p 

For  category  (iii)  -  Environment  Preference,  similar  reasoning  applies  (namely  that 
the  set  of  variables  represents  a  spanning  set  across  the  job  space,  and  that  each  variable 
is  independent  and  equally  weighted),  however  there  is  a  wrinkle  in  the  implementation. 
Candidate  preference  is  scored  as  a  single  value  between  two  presumably  related 
variables  (e.g.  indoor  and  outdoor)  thus  a  candidate  cannot  express  100  percent 
satisfaction  with  outdoor  work  and  at  the  same  time  100  percent  satisfaction  with 
indoor  work.  On  the  other  hand,  preliminary  data  used  for  job  attributes  indicates  that 
there  is  no  relationship  between  the  job  “DNA”  values  for  the  two  supposedly  related 
variables.  Although  this  implementation  appears  contradictory,  it  is  assumed  that  there 
is  sufficient  reason  for  this  approach.  It  is  therefore  recommended2  that  first  the  two  job 
attribute  values  are  converted  to  a  single  value  for  the  paired  variable,  by  taking  the 


1  It  is  assumed  in  the  text  that  all  percentage  preferences  are  converted  to  a  0-1  fraction. 

2  Based  on  two  factors  -  (i)  the  supposition  of  a  relationship  between  the  paired  variables  is  correct,  and 
(ii)  this  relationship  will  be  substantiated  in  the  job  DNA  data  as  that  data  is  refined  in  the  course  of  time. 
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appropriate  proportion  of  the  sum  of  the  two  values,  and  that  this  is  then  used  in  a 
normed  distance  calculation  as  described  for  category  (ii)  above.  This  value  (W3i  for 
job  i)  will  also  be  between  o  and  l. 

W3i=[Ee(Ce-jie)2]1/2  /  E* 

where  the  sum  is  over  the  E  paired  environment  meta-attributes  (variables), 
each  of  which  comprises  two  complementary  attributes  ei  and  e2 
ceis  the  candidate  preference  value  for  meta- attribute  e  (defined  as  =  ce2) 
jie  is  the  relative  job  “DNA”  value  for  job  i  for  meta-attribute  e,  calculated  as 
jie  =  jie2  /  (jiei  +  jie2)  (thus  between  o  and  l) 

The  three  weights  are  combined  to  get  a  ranking  value  for  each  job  i  as  follows: 

JobRankValuei  =  Wii  *  (W2i  +  W3i) 

and  the  jobs  can  then  be  ranked  in  order  of  lowest  JobRankValue  first. 

Finally,  a  simple  conversion  of  each  JobRankValuei  to  JOIN  Fit  Score: 

JobRanki  =  the  ranking  of  individual  JobRankValuei  from  o  to  79 

*  All  JobRanki  values  are  integers. 

JOIN  Fiti  =  too  -  JobRanki 

Now,  the  jobs  will  be  ranked  in  order  of  highest  ranked  JOIN  Fit  Score  first. 
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Example  Calculation  for  JOIN  JobRankValue; 
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Example  Calculation  for  JOIN  JobRankValue; 


DATA 


Variable 

Candidate 

Preference 

Jobl 

Job  Attributes  (DNA) 
Job2  Job3  Job  4 

Job5 

Category 

Air 

1 

1 

0 

0 

0 

0 

Surface 

3 

1 

1 

0 

0 

0 

Sub-surface 

4 

0 

0 

1 

1 

0 

Nuclear 

2 

0 

1 

1 

0 

0 

Space  Exploration 

none 

0 

0 

0 

0 

1 

Process 

Mechanical 

0.3 

1 

0 

1 

0 

0 

Intuitive 

0.2 

0 

1 

0 

1 

0 

Creative 

0.9 

0 

0 

0 

0 

1 

Precise 

0.5 

1 

1 

1 

1 

1 

Environment 

Indoor / 

0.7 

0 

1 

1 

1 

1 

Outdoor 

=(1-0.7)  =  0.3 

1 

0 

0 

0 

0.5 

Jolly  / 

0.5 

1 

1 

0 

1 

0 

Serious 

=(1-0.5)  =  0.5 

0 

1 

1 

0 

1 

CALCULATION 

Jobl 

W1  min(l,3)=l 

W2  SQRT((0.3-1)2  +  (0.2-0)2  +  (0.9-0)2  +  (0.5-1)2)  /  SQRT(4) 

=SQRT(0.34)/2  =  0.29 

W3  SQRT((l-0.3)2  +  (0-0.5)2)  /  SQRT(2) 

=SQRT(0.74)/1.4  =  0.61 
JobRankValue]  =  1  *  (0.29  +  0.61)  =0.90 

Job2 

W1  min(3,2)  =  2 

W2  SQRT((0. 3-0)2  +  (0.2-l)2  +  (0.9-0)2  +  (0.5-l)2)  /  SQRT(4) 

=SQRT(0.29)/2  =  0.27 

W3  SQRT((0-0.3)2  +  ([l/(l+l)]-0.5)2)  /  SQRT(2) 

=SQRT(0.3)/1.4  =  0.21 
JobRankValue2  =  2  *  (0.27  +  0.21)  =  0.96 
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Job3 

W1  min(4,2)  =  2 

W2  SQRT((0.3-1)2  +  (0.2-0)2  +  (0.9-0)2  +  (0.5-l)2)  /  SQRT(4) 

=SQRT(0.34)/2  =  0.29 

W3  SQRT((0-0.3)2  +  (l-0.5)2)  /  SQRT(2) 

=SQRT(0.34)/1.4  =  0.41 
JobRankValue3  =  2  *  (0.29  +  0.41)  =  1.40 

Job4 

W 1  min(4)  =  4 

W2  SQRT((0. 3-0)2  +  (0.2-l)2  +  (0.9-0)2  +  (0.5-l)2)  /  SQRT(4) 

=SQRT(0.29)/2  =  0.27 

W3  SQRT((0-0.3)2  +  (0-0.5)2)  /  SQRT(2) 

=SQRT(0.34)/1.4  =  0.41 
JobRankValue4  =  4  *  (0.27  +  0.41)  =  2.72 

Job5 

W1  min(99)  =  99 

W2  SQRT((0. 3-0)2  +  (0.2-0)2  +  (0.9-l)2  +  (0.5-l)2)  /  SQRT(4) 

=SQRT(1.06)/2  =  0.51 

W3  SQRT(([0.5/(l+0.5)]-0.3)2  +  (l-0.5)2)  /  SQRT(2) 

=SQRT(0.28)/1.4  =  0.38 
JobRankValue5  =  99  *  (0.51  +  0.38)  =  88.11  (!) 


RESULT 

The  final  Job  Ranking  is  therefore: 

(1)  Jobl  [0.90] 

(2)  Job2  [0.96] 

(3)  Job3  [1.40] 

(4)  Job4  [2.72] 

(5)  Job5  [88.11] 

To  present  the  Job  Ranking  by  JOIN  Fit  scores: 

(1)  Jobl  [100] 

(2)  Job2  [99] 

(3)  Job3  [98] 

(4)  Job4  [97] 

(5)  Job5  [96] 
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Pre-Inventory  Script 


You  are  being  invited  to  participate  in  the  development  of  a  Navy  classification  tool 
called  Jobs  and  Occupation  Interest  in  the  Navy  (JOIN)  developed  by  Navy  Personnel 
Research,  Studies,  and  Technology  (NPRST).  JOIN  is  a  computer-administered  survey 
that  intends  to  measure  recruits  interests  on  the  different  aspects  of  the  Navy  jobs.  Your 
responses  will  help  us  to  better  understand  the  interests  of  future  Sailors. 

This  is  a  longitudinal  study,  so  we  will  track  you  throughout  your  Navy  career  to 
participate  in  follow-up  studies.  Before  you  start,  you  will  be  asked  to  enter  you  Social 
Security  Number  (SSN).  If  you  wish  not  to  enter  your  SSN,  substitute  your  SSN  with 
999-99-9999.  At  this  moment,  please  raise  your  hand  if  you  do  not  wish  to  include  your 
SSN.  (Record  the  number  of  SSN  versus  no  SSN  participants) 

Throughout  the  test,  you  will  see  each  work  activity  twice.  Therefore,  the  items  may 
seem  repetitive.  However,  please  answer  each  question  independently.  Once  you  have 
completed  the  inventory,  please  pick  up  a  copy  of  the  Usability  Feedback  Survey,  and 
answer  the  questions  to  the  best  of  your  ability.  Participation  in  this  study  is  completely 
voluntary.  All  information  you  provide  will  be  kept  in  confidence,  and  will  only  be  used 
for  research  purposes.  Therefore,  your  responses  will  not  be  identified  individually.  In 
other  words,  your  involvement  with  this  survey  will  not  affect  your  Navy  career  in  any 
way. 

At  this  point,  does  anyone  have  any  questions  before  you  start?  (If  not)  You  may 
enter  your  Social  Security  Number  and  click  the  “Next”  button  to  start  the  survey. 
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JOIN  Usability  Feedback  Survey 
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JOIN  Usability  Feedback  Survey 


The  purpose  of  this  questionnaire  is  to  evaluate  the  quality  of  the  computer  software,  JOIN.  Your 
responses  will  be  analyzed  and  maintained  by  Navy  researchers  (NPRST/PERS-13).  All  responses  will  be 
held  in  confidence.  Information  provided  will  be  summarized  and  will  not  be  attributable  to  individuals. 
Your  participation  is  completely  voluntary.  Failure  to  respond  will  NOT  result  in  any  penalties  with  the 
exception  of  lack  of  representation  of  your  views  in  the  final  results  and  outcomes. 

Please  take  a  moment  to  reflect  back  on  JOIN  and  answer  the  following  questions: 

Usability 


Very  Bad 

Bad 

Neither 

Good 

Very  Good 

1 .  How  would  you  rate  JOIN  on 
the  following: 

a.  Ease  of  use... 

O 

O 

O 

O 

O 

b.  Instructions... 

O 

O 

O 

O 

O 

c.  Tutorials  (i.e.,  practice 
screens)... 

O 

O 

O 

O 

O 

d.  Visual  appeal. . . 

O 

o 

O 

O 

O 

e.  Holding  your  attention. . . 

O 

o 

O 

O 

O 

Worse 

Equal 

Better 

2.  Was  JOIN  worse  than,  the  same  as,  or  better  than 
paper  and  pencil  surveys  you  have  taken? 

O 

O 

O 

Information  &  Content 

Very  Bad 

Bad 

Neither 

Good 

Very  Good 

3.  How  well  did  the  headings  (i.e., 
textual  descriptions)  relate  to 
the  pictures  on  each  screen? 

O 

O 

O 

O 

O 

4.  How  well  could  you  understand 
what  was  being  represented  in 
each  picture? 

O 

o 

O 

O 

O 

5 .  How  did  JOIN  do  at  increasing 
your  knowledge  of  Navy  work 
areas,  styles,  and  activities? 

O 

o 

O 

O 

O 

Pictures  Only 

Text  Only 

Both 

6.  What  did  you  use  to  make  your 
decision  about  your  level  of 
interest  for  each  work  area  and 
activity? 

O 

O 

O 

7.  Did  the  pictures  provide  you  with  better 
information  than  the  text  alone? 

o 

No 

O 

Yes 
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Not  interested  50%  Very  interested 

8.  If  you  liked  one  picture  “very 
much”  out  of  the  three 
presented,  how  would  you 
respond? 

9.  If  you  liked  two  pictures  “very 
much”  out  of  the  three 
presented,  how  would  you 
respond? 

10.  If  you  liked  all  three  pictures 
“very  much”  how  would  you 
respond? 

Feedback  &  Opinions 

1 1 .  What  did  you  like  about  JOIN? 

12.  What  did  you  dislike  about 
JOIN? 


13.  Suggestions/ 

Recommendations? 


Learning  Environments 

Instructions:  Please  darken  to  circle  next  to  the  leaning  environment  you  would  most  prefer. 


14.  Computer  based  training  (CBT),  which  would  occur  from  a  personal  O 

computer  or  designated  locations  in  the  community. 

1 5 .  Traditional  classroom  training,  which  would  be  led  by  a  trained  instructor  O 

and  would  take  place  from  a  designated  location  in  the  community. 

General  Information 

16.  Before  coming  to  boot  camp,  how  many  hours  per  week  did  you  _ #  HRS 

spend  using  a  computer? 

17.  With  regard  to  your  computer  skills,  would  you  say  that  you  are. . . 

Not  experienced,  but  I  know  where  the  ON  switch  is  but  I  am  scared  to  O 

use  it 

Need  help  to  do  most  things  O 

Average  (email,  internet,  word  processing,  etc.)  O 


O  O  O 


O  O  O 


O  O  O 
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A  wide  variety  of  software  knowledge,  and  technical  expertise  O 

Very  experienced,  expert  user,  master  programmer,  and  I  can  build  a  O 

computer  from  scratch  in  less  than  2  minutes 

If  you  are  an  expert  user  please  list  any  special  abilities  (e.g.,  programming,  graphics,  etc.): 


18.  Please  list  any  musical  instruments  that  you  play: 

19.  Please  list  any  special  licenses  or  certification  credentials  you  may  have  (e.g.,  mortician, 
draftsman,  etc.): 


20.  Please  list  any  foreign  languages  that  you  speak  fluently: 


Appendix  G: 

First  Watch  Items  Used  for  JOIN  Predictive  Validation 
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First  Watch  Items  Used  for  JOIN  Predictive  Validation 


NQ  =  New  Sailor  Survey 
RQ  =  RTC  Survey 
AQ  =  Apprentice  School  Survey 
EQ  =  Exit  Survey 

NQ14  How  Successful  will  you  be  in  boot  camp? 

1 .  Not  applicable,  I  am  prior  military 

2.  I  will  do  better  than  most 

3.  I  will  do  about  as  well  as  everyone  else 

4.  I  will  not  do  as  well  as  most 

NQ  1 7  How  satisfied  about  time  with  classifier? 

1 .  Not  Applicable 

2.  Very  Satisfied 

3.  Satisfied 

4.  Neither  Satisfied  nor  dissatisfied 

5.  Dissatisfied 

6.  Very  Dissatisfied 

NQ19e  To  what  extend  was  each  of  the  following  explained  to  you: 

Job  you  were  assigned  at  classification? 

1 .  Very  Great  Extent 

2.  Great  Extent 

3.  Moderate  Extent 

4.  Slight  Extent 

5.  Not  at  All 

NQ19f  To  what  extend  was  each  of  the  following  explained  to  you: 

School  you  were  guaranteed  at  classification? 

1 .  Very  Great  Extent 

2.  Great  Extent 

3.  Moderate  Extent 

4.  Slight  Extent 

5.  Not  at  All 

NQ29  Preparation  for  RTC  received  from  my  recruiter  was: 

1 .  Not  Applicable 

2.  Excellent 

3.  Good 

4.  Satisfactory 

5.  Fair 

6.  Poor 


G-i 


RQ4b  The  Navy  jobs  available  to  you  at  classification: 

1 .  Very  Great  Extent 

2.  Great  Extent 

3.  Moderate  Extent 

4.  Slight  Extent 

5.  Not  at  All 

RQ4e  Job  you  were  assigned  at  classification: 

1 .  Very  Great  Extent 

2.  Great  Extent 

3.  Moderate  Extent 

4.  Slight  Extent 

5.  Not  at  All 

RQ4f  School  you  were  guaranteed  at  classification: 

1 .  Very  Great  Extent 

2.  Great  Extent 

3.  Moderate  Extent 

4.  Slight  Extent 

5.  Not  at  All 

RQ74a  Were  you  reclassified? 

1.  Yes 

2.  No,  but  I  wanted  to 

3.  No,  happy  with  rate 

AQ4  A-school  compared  with  expectations? 

1 .  Much  better  than  expected 

2.  Somewhat  better  than  expected 

3.  About  the  same  as  expected 

4.  Somewhat  worse  than  expected 

5.  Much  worse  than  expected 

AQ5  How  successful  were  you  in  A-School? 

1 .  Better  than  most 

2.  As  well  as  most 

3.  Worse  than  most 

AQ7  Satisfaction  with  current  rate? 

1.  Very  Satisfied 

2.  Satisfied 

3.  Neither 

4.  Dissatisfied 

5.  Very  Dissatisfied 
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AQ60  How  Navy  life  compared  with  expectations? 

1 .  Much  better  than  expected 

2.  Somewhat  better  than  expected 

3.  About  the  same  as  expected 

4.  Somewhat  worse  than  expected 

5.  Much  worse  than  expected 


EQ  63  How  Navy  Life  compared  with  expectations? 

1 .  Much  better  than  expected 

2.  Somewhat  better  than  expected 

3.  About  the  same  as  expected 

4.  Somewhat  worse  than  expected 

5.  Much  worse  than  expected 

EQ64f  Reasons  for  separation: 

Lack  of  motivation  or  boredom 

1.  Yes 

2.  No 
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Appendix  H: 

Internal  Consistencies  of  Work  Activity  Items  on 
JOIN  1.0  Delta  and  JOIN  l.Ole. 
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Internal  consistencies  of  work  activity  items  on 
JOIN  1.0  Delta  and  JOIN  l.Ole. 


Item- 

Item- 

Item- 

Work  Activities 

Triplets 

Couplets 

Couplets 

(Process-Content  Pairs) 

(N  =  287) 

(N  =  4,348) 

(N  =  300) 

Analyze  Communications 

0.8931 

0.8247 

0.8336 

Analyze  Data 

0.8706 

0.8389 

0.8305 

Analyze  Documents 

0.8613 

0.8481 

0.8494 

Direct  Aircraft 

0.9270 

0.8890 

0.8864 

Direct  Emergency  Response 

0.8974 

0.8176 

0.8286 

Maintain  Documents 

0.9068 

0.8430 

0.8411 

Maintain  Electrical  Equipment 

0.9357 

0.8753 

0.8744 

Maintain  Electronic  Equipment 

0.9128 

0.8625 

0.8684 

Maintain  Facilities 

0.8719 

0.7942 

0.7858 

Maintain  Mechanical  Equipment 

0.8687 

0.8496 

0.8373 

Maintain  Security 

0.8984 

0.9007 

0.9241 

Maintain  Supplies 

0.8842 

0.8593 

0.8253 

Maintain  Weapons 

0.8249 

0.8751 

0.8945 

Make  Communications 

0.8589 

0.7678 

0.7757 

Make  Documents 

0.8814 

0.8727 

0.8961 

Make  Facilities 

0.9452 

0.9139 

0.9065 

Make  Mechanical  Equipment 

0.9194 

0.8919 

0.8518 

Operate  Electrical  Equipment 

0.8980 

0.8434 

0.8528 

Operate  Electronic  Equipment 

0.8886 

0.8058 

0.8354 

Operate  Facilities 

0.9110 

0.8478 

0.8836 

Operate  Mechanical  Equipment 

0.8302 

0.7921 

0.7917 

Operate  Office  Equipment 

0.9342 

0.9133 

0.9117 

Operate  Weapons 

0.9185 

0.9094 

0.9195 

Respond  to  Emergencies 

0.9189 

0.8609 

0.8429 

Serve  Customers 

0.9073 

0.8419 

0.8372 

Train  People 

0.9053 

0.8651 

0.8584 

Group  Maximum 

0.9452 

0.9139 

0.9241 

Group  Minimum 

0.8249 

0.7678 

0.7757 

Individual  Group  Average 

0.8950 

0.8540 

0.8555 

H-i 


Appendix  I: 

Changes  in  Internal  Consistencies  of  Work  Activity  Items 
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Changes  in  internal  consistencies  of  work  activity  items  -  JOIN  1.0  Delta 
versus  JOIN  l.Ole,  JOINl.O  Delta  versus  JOIN  l.Ole  (300  random  sample), 
and  JOIN  l.Ole  (complete  sample)  versus  JOIN  l.Ole  (300  random  sample). 


Work  Activities 

A  in  Alpha 

A  in  Alpha 

A  in  Alpha 

(Process-Content  Pairs) 

(287  vs.  4,348) 

(287  vs.  300) 

(4,348  -  300) 

Analyze  Communications 

0.0684 

0.0595 

(0.0089) 

Analyze  Data 

0.0317 

0.0401 

0.0084 

Analyze  Documents 

0.0132 

0.0119 

0.0013 

Direct  Aircraft 

0.0380 

0.0406 

0.0026 

Direct  Emergency  Response 

0.0798 

0.0688 

(0.0110) 

Maintain  Documents 

0.0638 

0.0657 

0.0019 

Maintain  Electrical  Equipment 

0.0604 

0.0613 

0.0009 

Maintain  Electronic  Equipment 

0.0503 

0.0444 

(0.0059) 

Maintain  Facilities 

0.0777 

0.0861 

0.0084 

Maintain  Mechanical  Equipment 

0.0191 

0.0314 

0.0123 

Maintain  Security 

(0.0023) 

(0.0257) 

(0.0234) 

Maintain  Supplies 

0.0249 

0.0589 

0.0340 

Maintain  Weapons 

(0.0502) 

(0.0696) 

(0.0194) 

Make  Communications 

0.0911 

0.0832 

(0.0079) 

Make  Documents 

0.0087 

(0.0147) 

(0.0234) 

Make  Facilities 

0.0313 

0.0387 

0.0074 

Make  Mechanical  Equipment 

0.0275 

0.0676 

0.0401 

Operate  Electrical  Equipment 

0.0546 

0.0452 

(0.0094) 

Operate  Electronic  Equipment 

0.0828 

0.0532 

(0.0296) 

Operate  Facilities 

0.0632 

0.0274 

(0.0358) 

Operate  Mechanical  Equipment 

0.0381 

0.0385 

0.0004 

Operate  Office  Equipment 

0.0209 

0.0225 

0.0016 

Operate  Weapons 

0.0091 

(0.0010) 

(0.0101) 

Respond  to  Emergencies 

0.0580 

0.0760 

0.0180 

Serve  Customers 

0.0654 

0.0701 

0.0047 

Train  People 

0.0402 

0.0469 

0.0067 

Group  Maximum  Change 

0.0911 

0.0861 

0.0401 

Group  Minimum  Change 

0.0023 

0.0010 

0.0004 

Group  Average  Change 

0.0450 

0.0480 

0.0128 

Note.  Values  in  Parentheses  Represent  Negative  Changes. 
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Appendix  J: 

Internal  Consistencies  of  Work  Activity  Items  Among 
Racial  Subgroups  (JOIN  l.Ole) 
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Internal  Consistencies  of  Work  Activity  Items  Among 
Racial  Subgroups  (JOIN  l.Ole) 


Work  Activities 

Asian 

Black 

Hispanic 

White 

Others 

(Process-Content  Pairs) 

z 

n 

VD 

o 

1 — ' 

(N=687) 

(N=625) 

(N= 2,490) 

(N=257) 

Analyze  Communications 

0.8420 

0.7962 

0.8288 

0.8332 

0.7587 

Analyze  Data 

0.9039 

0.8106 

0.8322 

0.8446 

0.8254 

Analyze  Documents 

0.8681 

0.8453 

0.8384 

0.8443 

0.8370 

Direct  Aircraft 

0.8670 

0.8895 

0.8881 

0.8942 

0.8597 

Direct  Emergency  Response 

0.8318 

0.7865 

0.8234 

0.8289 

0.7482 

Maintain  Documents 

0.8213 

0.8424 

0.8251 

0.8322 

0.8147 

Maintain  Electrical 

0.8521 

0.8999 

0.8589 

0.8734 

0.8733 

Equipment 

Maintain  Electronic 

0.8845 

0.8240 

0.8505 

0.8747 

0.8594 

Equipment 

Maintain  Facilities 

0.7768 

0.7819 

0.8137 

0.7939 

0.7744 

Maintain  Mechanical 

0.8085 

0.8568 

0.8243 

0.8583 

0.8476 

Equipment 

Maintain  Security 

0.9087 

0.8889 

0.8961 

0.9064 

0.8689 

Maintain  Supplies 

0.8391 

0.8278 

0.8800 

0.8509 

0.9025 

Maintain  Weapons 

0.8629 

0.8888 

0.8669 

0.8744 

0.8425 

Make  Communications 

0.7204 

0.7845 

0.7438 

0.7674 

0.7231 

Make  Documents 

0.8871 

0.8570 

0.8715 

0.8672 

0.8823 

Make  Facilities 

0.9223 

0.8920 

0.8958 

0.9265 

0.8812 

Make  Mechanical  Equipment 

0.8512 

0.8803 

0.8888 

0.9029 

0.8379 

Operate  Electrical 

0.8262 

0.8124 

0.8317 

0.8587 

0.8140 

Equipment 

Operate  Electronic 

0.8459 

0.7706 

0.7551 

0.8283 

0.8081 

Equipment 

Operate  Facilities 

0.8788 

0.8269 

0.8446 

0.8407 

0.8402 

Operate  Mechanical 

0.7055 

0.8123 

0.7871 

0.7962 

0.7583 

Equipment 

Operate  Office  Equipment 

0.9046 

0.9033 

0.8967 

0.9061 

0.9252 

Operate  Weapons 

0.9178 

0.9040 

0.9290 

0.9044 

0.8893 

Respond  to  Emergencies 

0.8391 

0.8639 

0.8692 

0.8604 

0.8140 

Serve  Customers 

0.8358 

0.8205 

0.8361 

0.8252 

0.8215 

Train  People 

0.8179 

0.8367 

0.8860 

0.8703 

0.8638 

Group  Maximum 

0.9223 

0.9040 

0.9290 

0.9265 

0.9252 

Group  Minimum 

0.7055 

0.7706 

0.7438 

0.7674 

0.7231 

Individual  Group  Average 

0.8469 

0.8424 

0.8485 

0.8563 

0.8335 
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Appendix  K: 

Significance  Tests  on  Alpha  Reliability  of 
Work  Activity  Items 
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Significance  Tests  on  Alpha  Reliability  of 
Work  Activity  Items 


Subgroup  Comparison 


(JOIN  l.Ole) 

*7 

a 

S 

z 

Male 

3,307 

0.8473 

Female 

942 

0.8541 

1.0466 

0.4938 

Asian-American 

190 

0.8463 

Caucasian-American 

2,490 

0.8563 

1.0654 

1.1174 

African-American 

687 

0.8424 

Caucasian-American 

2,490 

0.8563 

1.0967 

1.6283 

Hispanic-American 

625 

0.8485 

Caucasian-American 

2,490 

0.8563 

1.0542 

0.9322 

Asian-American 

190 

0.8463 

African-American 

687 

0.8424 

0.9714 

-0.2681 

African-American 

687 

0.8424 

Hispanic-American 

625 

0.8485 

1.0403 

0.3484 

Asian-American 

191 

0.8469 

Hispanic-American 

625 

0.8485 

1.0106 

0.0927 

Note:  za/2  =  1.96 

S  =  [l-Pi)l  (I-P2) 


Pi=  «i 
P2=  a 2 

I  f  za/2  >  Z>  Za/2\  pi  =  P2 
If  Z>  Za/2,  pi  <  P2 
If  Z  >  -Za/2,  pi  >  P2 
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Appendix  L: 

Internal  Consistencies  of  Work  Activity  Items  Between 
Gender  Subgroups  (JOIN  l.Ole) 


L-o 


Internal  Consistencies  of  Work  Activity  Items  Between 
Gender  Subgroups  (JOIN  l.Ole) 


Work  Activities 

Male 

Female 

(Process-Content  PAIRS) 

(N  =  3,307) 

(N  =  942) 

Analyze  Communications 

0.8201 

0.8266 

Analyze  Data 

0.8355 

0.8463 

Analyze  Documents 

0.8404 

0.8406 

Direct  Aircraft 

0.8881 

0.8942 

Direct  Emergency  Response 

0.8152 

0.8221 

Maintain  Documents 

0.8270 

0.8293 

Maintain  Electrical  Equipment 

0.8621 

0.8883 

Maintain  Electronic  Equipment 

0.8583 

0.8483 

Maintain  Facilities 

0.8004 

0.7722 

Maintain  Mechanical  Equipment 

0.8346 

0.8646 

Maintain  Security 

0.9005 

0.8979 

Maintain  Supplies 

0.8613 

0.8515 

Maintain  Weapons 

0.8598 

0.8796 

Make  Communications 

0.7538 

0.7784 

Make  Documents 

0.8570 

0.8834 

Make  Facilities 

0.9095 

0.9161 

Make  Mechanical  Equipment 

0.8862 

0.8765 

Operate  Electrical  Equipment 

0.8435 

0.8329 

Operate  Electronic  Equipment 

0.8132 

0.7904 

Operate  Facilities 

0.8430 

0.8589 

Operate  Mechanical  Equipment 

0.7683 

0.8083 

Operate  Office  Equipment 

0.9021 

0.9244 

Operate  Weapons 

0.8985 

0.9078 

Respond  to  Emergencies 

0.8525 

0.8853 

Serve  Customers 

0.8333 

0.8179 

Train  People 

0.8656 

0.8644 

Group  Maximum 

0.9095 

0.9244 

Group  Minimum 

0.7538 

0.7722 

Individual  Group  Average 

0.8473 

0.8541 
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Appendix  M: 

Correlation  Analyses  on  JOIN  Fit  Scores  and  First  Watch 

Items 
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Correlation  Analyses  on  JOIN  Fit  Scores  and  First  Watch 

Items 


NQ14  How  successful  will  you  be  in  boot  camp 

Correlation  Coefficient 
Sig.  (2-tailed) 

N 

0.051 

0.032 

1773 

NQ17  How  satisfied  about  time  with  classifier 

Correlation  Coefficient 

-0.034 

Sig.  (2-tailed) 

0.150 

N 

1765 

NQ19E  J  ob  you  were  assigned  at  classification 

Correlation  Coefficient 

0.042 

Sig.  (2-tailed) 

0.076 

N 

1766 

NQ19F  School  you  were  guaranteed  at  classification 

Correlation  Coefficient 

0.059 

Sig.  (2-tailed) 

0.012 

N 

1767 

NQ29  Preparation  for  RTC  received  from  my 
recruiter  was 

Correlation  Coefficient 

-0.049 

Sig.  (2-tailed) 

0.040 

N 

1768 

RQ4B  The  Navy  jobs  available  to  you  at  classification 

Correlation  Coefficient 

0.065 

Sig.  (2-tailed) 

0.021 

N 

1256 

RQ4E  J  ob  you  were  assigned  at  classification 

Correlation  Coefficient 

0.059 

Sig.  (2-tailed) 

0.036 

N 

1252 

RQ4F  School  you  were  guaranteed  at  classification 

Correlation  Coefficient 

0.076 

Sig.  (2-tailed) 

0.007 

N 

1251 

RQ74A  Were  you  reclassified 

Correlation  Coefficient 

-0.179 

Sig.  (2-tailed) 

0.000 

N 

1149 

AQ7  Satisfaction  with  current  rate 

Correlation  Coefficient 

0.149 

Sig.  (2-tailed) 

0.009 

N 

311 

Note.  Reverse  coding  was  applied  prior  to  data  analyses. 
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Appendix  N: 

Post  hoc  Analyses  on  Usability  Feedback  Survey 


N-o 


Post  hoc  Analyses  on  Usability  Feedback  Survey  Based  on 
the  Usage  of  Textual  Presentation,  Visual  Presentation  or 
the  Combination  of  Both  (JOIN  l.Ole) 


Dependent 

Variable 

(I)  Q6  What  did  you  used  to 
make  your  decision  about  your 
level  of  interest  for  each  work 
area  and  activity? 

Mean 

Difference 

(I-J) 

Std. 

Error 

Sig. 

Q1A  Ease  of 

Tukey  HSD 

1  Pictures  Only 

2 

Text  Only 

.02 

.113 

.983 

Use 

Rating 

3 

Both 

-.14 

.061 

.054 

2  Text  Only 

1 

Pictures  Only 

-.02 

.113 

.983 

3 

Both 

-.16 

.100 

.239 

3  Both 

1 

Pictures  Only 

.14 

.061 

.054 

2 

Text  Only 

.16 

.100 

.239 

Bonferroni 

1  Pictures  Only 

2 

Text  Only 

.02 

.113 

1.00 

n 

3 

Both 

-.14 

.061 

u 

.062 

2  Text  Only 

1 

Pictures  Only 

-.02 

.113 

1.00 

n 

3 

Both 

-.16 

.100 

u 

.319 

3  Both 

1 

Pictures  Only 

.14 

.061 

.062 

2 

Text  Only 

.16 

.100 

.319 

Games-Howell 

1  Pictures  Only 

2 

Text  Only 

.02 

.119 

.985 

3 

Both 

-.14 

.068 

.097 

2  Text  Only 

1 

Pictures  Only 

-.02 

.119 

.985 

3 

Both 

-.16 

.103 

.269 

3  Both 

1 

Pictures  Only 

.14 

.068 

.097 

2 

Text  Only 

.16 

.103 

.269 

Q1B 

Tukey  HSD 

1  Pictures  Only 

2 

Text  Only 

.03 

.113 

.969 

Instructi 

ons 

Rating 

3 

Both 

-.23(*) 

.062 

.001 

2  Text  Only 

1 

Pictures  Only 

-.03 

.113 

.969 

3 

Both 

-,25(*) 

.100 

.032 

3  Both 

1 

Pictures  Only 

,23(*) 

.062 

.001 

2 

Text  Only 

.  25(* ) 

.100 

.032 

Bonferroni 

1  Pictures  Only 

2 

Text  Only 

.03 

.113 

1.00 

n 

3 

Both 

-.23(*) 

.062 

u 

.001 

2  Text  Only 

1 

Pictures  Only 

-.03 

.113 

1.00 

n 

3 

Both 

-,25(*) 

.100 

u 

.035 

3  Both 

1 

Pictures  Only 

.23(*) 

.062 

.001 

2 

Text  Only 

.  25(* ) 

.100 

.035 

N-i 


(I)  Q6  What  did  you  used  to 
make  your  decision  about  your 

Mean 

Dependent 

level  of  interest  for  each  work 

Difference 

Std. 

Variable 

area  and  activity? 

(I-J) 

Error 

Sig. 

Q1B 

Games-Howell 

1  Pictures  Only 

2  Text  Only 

.03 

.116 

.971 

1  nstructions 
Rating 

3  Both 

-.23(*) 

.066 

.002 

2  Text  Only 

1  Pictures  Only 

-.03 

.116 

.971 

3  Both 

-.25(*) 

.101 

.041 

3  Both 

1  Pictures  Only 

.23(*) 

.066 

.002 

2  Text  Only 

,25(*) 

.101 

.041 

QIC  Tutorials 

Tukey  HSD 

1  Pictures  Only 

2  Text  Only 

.17 

.140 

.442 

Rating 

3  Both 

-.16 

.076 

.104 

2  Text  Only 

1  Pictures  Only 

-.17 

.140 

.442 

3  Both 

-•33(*) 

.124 

.024 

3  Both 

1  Pictures  Only 

.16 

.076 

.104 

2  Text  Only 

■  33(*) 

.124 

.024 

Bonferroni 

1  Pictures  Only 

2  Text  Only 

.17 

.140 

.669 

3  Both 

-.16 

.076 

.125 

2  Text  Only 

1  Pictures  Only 

-.17 

.140 

.669 

3  Both 

-■33(*) 

.124 

.026 

3  Both 

1  Pictures  Only 

.16 

.076 

.125 

2  Text  Only 

■  33(*) 

.124 

.026 

Games-Howell 

1  Pictures  Only 

2  Text  Only 

.17 

.152 

.505 

3  Both 

-.16 

.077 

.110 

2  Text  Only 

1  Pictures  Only 

-.17 

.152 

.505 

3  Both 

-.33 

.138 

.056 

3  Both 

1  Pictures  Only 

.16 

.077 

.110 

2  Text  Only 

.33 

.138 

.056 

Q1D  Visual 

Tukey  HSD 

1  Pictures  Only 

2  Text  Only 

-.02 

.138 

.983 

Appeal 

Rating 

3  Both 

-.05 

.075 

.801 

2  Text  Only 

1  Pictures  Only 

.02 

.138 

.983 

3  Both 

-.02 

.123 

.980 

3  Both 

1  Pictures  Only 

.05 

.075 

.801 

2  Text  Only 

.02 

.123 

.980 

Bonferroni 

1  Pictures  Only 

2  Text  Only 

-.02 

.138 

1.00 

0 

1.00 

0 

1.00 

0 

1.00 

0 

1.00 

0 

1.00 

3  Both 

-.05 

.075 

2  Text  Only 

1  Pictures  Only 

.02 

.138 

3  Both 

-.02 

.123 

3  Both 

1  Pictures  Only 

.05 

.075 

2  Text  Only 

.02 

.123 

N-2 


(I)  Q6  What  did  you  used  to 
make  your  decision  about  your 

Mean 

Dependent 

level  of  interest  for  each  work 

Difference 

Std. 

Variable 

area  and  activity? 

(I-J) 

Error 

Sig. 

0 

Q1D  Visual 

Games-Howell 

1  Pictures  Only 

2  Text  Only 

-.02 

.141 

.984 

Appeal  Rating 

3  Both 

-.05 

.078 

.815 

2  Text  Only 

1  Pictures  Only 

.02 

.141 

.984 

3  Both 

-.02 

.124 

.980 

3  Both 

1  Pictures  Only 

.05 

.078 

.815 

2  Text  Only 

.02 

.124 

.980 

Q1E  Holding 
your  Attention 

Tukey  HSD 

1  Pictures  Only 

2  Text  Only 

.40 

.177 

.064 

Rating 

3  Both 

-.10 

.097 

.539 

2  Text  Only 

1  Pictures  Only 

-.40 

.177 

.064 

3  Both 

-,50(*) 

.157 

.004 

3  Both 

1  Pictures  Only 

.10 

.097 

.539 

2  Text  Only 

■son 

.157 

.004 

Bonferroni 

1  Pictures  Only 

2  Text  Only 

.40 

.177 

.075 

3  Both 

-.10 

.097 

.867 

2  Text  Only 

1  Pictures  Only 

-.40 

.177 

.075 

3  Both 

-,50(*) 

.157 

.004 

3  Both 

1  Pictures  Only 

.10 

.097 

.867 

2  Text  Only 

son 

.157 

.004 

Games-Howell 

1  Pictures  Only 

2  Text  Only 

.40 

.184 

.084 

3  Both 

-.10 

.105 

.588 

2  Text  Only 

1  Pictures  Only 

-.40 

.184 

.084 

3  Both 

-.50H 

.160 

.008 

3  Both 

1  Pictures  Only 

.10 

.105 

.588 

2  Text  Only 

■so  n 

.160 

.008 

Q2  Comparison 

Tukey  HSD 

1  Pictures  Only 

2  Text  Only 

.04 

.067 

.800 

to  Paper- 
Pencil 

Format 

3  Both 

-.06 

.037 

.203 

2  Text  Only 

1  Pictures  Only 

-.04 

.067 

.800 

3  Both 

-.11 

.060 

.178 

3  Both 

1  Pictures  Only 

.06 

.037 

.203 

2  Text  Only 

.11 

.060 

.178 

Bonferroni 

1  Pictures  Only 

2  Text  Only 

.04 

.067 

1.000 

3  Both 

-.06 

.037 

.265 

2  Text  Only 

1  Pictures  Only 

-.04 

.067 

1.000 

3  Both 

-.11 

.060 

.228 

3  Both 

1  Pictures  Only 

.06 

.037 

.265 

2  Text  Only 

.11 

.060 

.228 

(I)  Q6  What  did  you  used  to 


make  your  decision  about  your 

Mean 

Dependent 

level  of  interest  for  each  work 

Difference 

Std. 

Variable 

area  and  activity? 

(I-J) 

Error 

Sig. 

Q2 

Games-Howell 

1  Pictures  Only 

2  Text  Only 

.04 

.082 

.860 

Comparison  to 

Paper-Pencil 

Format 

3  Both 

-.06 

.041 

.275 

2  Text  Only 

1  Pictures  Only 

-.04 

.082 

.860 

3  Both 

-.11 

.073 

.327 

3  Both 

1  Pictures  Only 

.06 

.041 

.275 

2  Text  Only 

.11 

.073 

.327 

Q3  How  well 
did  the 
descriptions 
relate  to  the 

Tukey  HSD 

1  Pictures  Only 

2  Text  Only 

.03 

.107 

.958 

pictures  on  each 
screen? 

3  Both 

-.17(*) 

.058 

.008 

2  Text  Only 

1  Pictures  Only 

-.03 

.107 

.958 

3  Both 

-.20 

.095 

.082 

3  Both 

1  Pictures  Only 

■17  n 

.058 

.008 

2  Text  Only 

.20 

.095 

.082 

Bonferroni 

1  Pictures  Only 

2  Text  Only 

.03 

.107 

1.000 

3  Both 

-,17(*) 

.058 

.008 

2  Text  Only 

1  Pictures  Only 

-.03 

.107 

1.000 

3  Both 

-.20 

.095 

.097 

3  Both 

1  Pictures  Only 

- 1 7  (*) 

.058 

.008 

2  Text  Only 

.20 

.095 

.097 

Games-Howell 

1  Pictures  Only 

2  Text  Only 

.03 

.129 

.970 

3  Both 

-.17(*) 

.066 

.025 

2  Text  Only 

1  Pictures  Only 

-.03 

.129 

.970 

3  Both 

-.20 

.114 

.184 

3  Both 

1  Pictures  Only 

■17(*) 

.066 

.025 

2  Text  Only 

.20 

.114 

.184 

Q4  How  well 

Tukey  HSD 

1  Pictures  Only 

2  Text  Only 

.07 

.108 

.798 

could  you 
understand 
what  was 
being 

represented 
in  each 
picture? 


3  Both 

-.21  (*) 

.059 

.001 

2  Text  Only 

1  Pictures  Only 

-.07 

.108 

.798 

3  Both 

-■27  n 

.096 

.012 

3  Both 

1  Pictures  Only 

■2i  n 

.059 

.001 

2  Text  Only 

-27  (*) 

.096 

.012 

(I)  Q6  What  did  you  used  to 
make  your  decision  about  your  Mean 


Dependent  level  of  interest  for  each  work  Difference  Std. 

Variable  area  and  activity?  (I-J)  Error  Sig. 


Q4  How  well 

Bonferroni 

1  Pictures  Only 

2  Text  Only 

.07 

.108 

1.000 

could  you 
understand 

what  was 
being 

represented 
in  each 
picture? 

3  Both 

-,21(*) 

.059 

.002 

2  Text  Only 

1  Pictures  Only 

-.07 

.108 

1.000 

3  Both 

-■27  (*) 

.096 

.013 

3  Both 

1  Pictures  Only 

.21  n 

.059 

.002 

2  Text  Only 

■27(*) 

.096 

.013 

Games-Howell 

1  Pictures  Only 

2  Text  Only 

.07 

.110 

.803 

3  Both 

-,21(*) 

.065 

.005 

2  Text  Only 

1  Pictures  Only 

-.07 

.110 

.803 

3  Both 

-■27  n 

.094 

.014 

3  Both 

1  Pictures  Only 

.21  n 

.065 

.005 

2  Text  Only 

■27  (*) 

.094 

.014 

Q5  How  Did 

JOIN  do  at 

Tukey  HSD 

1  Pictures  Only 

2  Text  Only 

,48(*) 

.151 

.005 

increasing  your 
knowledge  of 
Navy  work 
areas,  styles, 
and  activities? 

3  Both 

-.17 

.081 

.082 

2  Text  Only 

1  Pictures  Only 

-.480 

.151 

.005 

3  Both 

-.650 

.134 

.000 

3  Both 

1  Pictures  Only 

.17 

.081 

.082 

2  Text  Only 

.650 

.134 

.000 

Bonferroni 

1  Pictures  Only 

2  Text  Only 

.480 

.151 

.005 

3  Both 

-.17 

.081 

.097 

2  Text  Only 

1  Pictures  Only 

-.480 

.151 

.005 

3  Both 

-.650 

.134 

.000 

3  Both 

1  Pictures  Only 

.17 

.081 

.097 

2  Text  Only 

.650 

.134 

.000 

Games-Howell 

1  Pictures  Only 

2  Text  Only 

.480 

.161 

.011 

3  Both 

-.17 

.087 

.113 

2  Text  Only 

1  Pictures  Only 

-,48(*) 

.161 

.011 

3  Both 

-.650 

.143 

.000 

3  Both 

1  Pictures  Only 

.17 

.087 

.113 

2  Text  Only 

.650 

.143 

.000 

(I)  Q6  What  did  you  used  to 
make  your  decision  about  your 

Mean 

Dependent 

level  of  interest  for  each  work 

Difference 

Std. 

Variable 

area  and  activity? 

(I-J) 

Error 

Sig. 

Q8  If  you  liked 

Tukey  HSD 

1  Pictures  Only 

2  Text  Only 

.14 

.113 

.452 

ONE 

picture 
"very  much" 
out  of  the 

three 

presented, 
how  would 

you 

respond? 

3  Both 

.02 

.061 

.954 

2  Text  Only 

1  Pictures  Only 

-.14 

.113 

.452 

3  Both 

-.12 

.100 

.470 

3  Both 

1  Pictures  Only 

-.02 

.061 

.954 

2  Text  Only 

.12 

.100 

.470 

Bonferroni 

1  Pictures  Only 

2  Text  Only 

.14 

.113 

.688 

3  Both 

.02 

.061 

1.000 

2  Text  Only 

1  Pictures  Only 

-.14 

.113 

.688 

3  Both 

-.12 

.100 

.723 

3  Both 

1  Pictures  Only 

-.02 

.061 

1.000 

2  Text  Only 

.12 

.100 

.723 

Games-Howell 

1  Pictures  Only 

2  Text  Only 

.14 

.114 

.461 

3  Both 

.02 

.062 

.954 

2  Text  Only 

1  Pictures  Only 

-.14 

.114 

.461 

3  Both 

-.12 

.101 

.480 

3  Both 

1  Pictures  Only 

-.02 

.062 

.954 

2  Text  Only 

.12 

.101 

.480 

Q9  If  you  liked 

Tukey  HSD 

1  Pictures  Only 

2  Text  Only 

.02 

.089 

.983 

TWO 

pictures 
"very  much" 
out  of  the 
three 

presented, 
how  would 

you 

respond? 

3  Both 

-.03 

.048 

.847 

2  Text  Only 

1  Pictures  Only 

-.02 

.089 

.983 

3  Both 

-.04 

.079 

.854 

3  Both 

1  Pictures  Only 

.03 

.048 

.847 

2  Text  Only 

.04 

.079 

.854 

Bonferroni 

1  Pictures  Only 

2  Text  Only 

.02 

.089 

1.000 

3  Both 

-.03 

.048 

1.000 

2  Text  Only 

1  Pictures  Only 

-.02 

.089 

1.000 

3  Both 

-.04 

.079 

1.000 

3  Both 

1  Pictures  Only 

.03 

.048 

1.000 

2  Text  Only 

.04 

.079 

1.000 

N-6 


(I)  Q6  What  did  you  used  to 
make  your  decision  about  your  Mean 


Dependent 

level  of  interest  for  each  work 

Difference 

Std. 

Variable 

area  and  activity? 

(I-J) 

Error 

Sig. 

Q9  If  you  liked 

Games-Howell 

1  Pictures  Only 

2  Text  Only 

.02 

.088 

.982 

TWO 

pictures 
"very  much" 
out  of  the 
three 

presented, 
how  would 

you 

respond? 

3  Both 

-.03 

.049 

.852 

2  Text  Only 

1  Pictures  Only 

-.02 

.088 

.982 

3  Both 

-.04 

.077 

.847 

3  Both 

1  Pictures  Only 

.03 

.049 

.852 

2  Text  Only 

.04 

.077 

.847 

Q10  If  you  liked 

Tukey  HSD 

1  Pictures  Only 

2  Text  Only 

.06 

.069 

.690 

ALL 

THREE 
pictures 
"very 
much", 
how  would 

you 

respond? 

3  Both 

.00 

.038 

.995 

2  Text  Only 

1  Pictures  Only 

-.06 

.069 

.690 

3  Both 

-.06 

.061 

.588 

3  Both 

1  Pictures  Only 

.00 

.038 

.995 

2  Text  Only 

.06 

.061 

.588 

Bonferroni 

1  Pictures  Only 

2  Text  Only 

.06 

.069 

1.000 

3  Both 

.00 

.038 

1.000 

2  Text  Only 

1  Pictures  Only 

-.06 

.069 

1.000 

3  Both 

-.06 

.061 

.978 

3  Both 

1  Pictures  Only 

.00 

.038 

1.000 

2  Text  Only 

.06 

.061 

.978 

Games-Howell 

1  Pictures  Only 

2  Text  Only 

.06 

.073 

.717 

3  Both 

.00 

.040 

.996 

2  Text  Only 

1  Pictures  Only 

-.06 

.073 

.717 

3  Both 

-.06 

.064 

.618 

3  Both 

1  Pictures  Only 

.00 

.040 

.996 

2  Text  Only 

.06 

.064 

.618 

N-7 


(I)  Q6  What  did  you  used  to 
make  your  decision  about  your  Mean 


Dependent 

level  of  interest  for  each  work 

Difference 

Std. 

Variable 

area  and  activity? 

(I-J) 

Error 

Sig. 

Q17A  With 
regard  to  your 

Tukey  HSD 

1  Pictures  Only 

2  Text  Only 

-.28 

.123 

.063 

computer  skills, 
would  you  say 
that  you  are.... 

3  Both 

-.16 

.068 

.053 

2  Text  Only 

1  Pictures  Only 

.28 

.123 

.063 

3  Both 

.12 

.109 

.514 

3  Both 

1  Pictures  Only 

.16 

.068 

.053 

2  Text  Only 

-.12 

.109 

.514 

Bonferroni 

1  Pictures  Only 

2  Text  Only 

-.28 

.123 

.073 

3  Both 

-.16 

.068 

.061 

2  Text  Only 

1  Pictures  Only 

.28 

.123 

.073 

3  Both 

.12 

.109 

.815 

3  Both 

1  Pictures  Only 

.16 

.068 

.061 

2  Text  Only 

-.12 

.109 

.815 

Games-Howell 

1  Pictures  Only 

2  Text  Only 

-.28 

.118 

.054 

3  Both 

-,16(*) 

.066 

.045 

2  Text  Only 

1  Pictures  Only 

.28 

.118 

.054 

3  Both 

.12 

.104 

.488 

3  Both 

1  Pictures  Only 

■16(*) 

.066 

.045 

2  Text  Only 

-.12 

.104 

.488 

*  The  mean  difference  is  significant  at  the  .05  level. 
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Appendix  O: 

Stepwise  Regression  Analysis 


O-o 


Stepwise  Regression  Analysis  on  How  JOIN  l.Ole  Holds 

the  Attention  of  Participants 


Model 

R 

R 

Square 

Adjusted 

R  Square 

Std.  Error 
of  the 

Estimate 

Change  Statistics 

R  Square 

F 

Sig.  F 

Change 

Change 

dfl 

df2 

Change 

1 

.552(a) 

.305 

.304 

.852  .305 

370.177 

1 

845 

.000 

2 

.625(b) 

.390 

.389 

.799  .086 

118.328 

1 

844 

.000 

3 

.635(c) 

.403 

.401 

.791  .013 

17.652 

1 

843 

.000 

4 

.641(d) 

.410 

.407 

.786  .008 

10.913 

1 

842 

.001 

Note,  a  Predictors:  (Constant),  Q1D  Visual  Appeal  Rating 

b  Predictors:  (Constant),  Q1D  Visual  Appeal  Rating,  Q5  How  Did  J  01 N  do  at  increasing  your  knowledge  of  Navy  work 
areas,  styles,  and  activities? 

c  Predictors:  (Constant),  Q1D  Visual  Appeal  Rating,  Q5  How  Did  J  01 N  do  at  increasing  your  knowledge  of  Navy  work 
areas,  styles,  and  activities?,  Q1B  Instructions  Rating 

d  Predictors:  (Constant),  Q1D  Visual  Appeal  Rating,  Q5  How  Did  J  01 N  do  at  increasing  your  knowledge  of  Navy  work 
areas,  styles,  and  activities?,  Q1B  Instructions  Rating,  02  Comparison  to  Paper-Pencil  Format 
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Distribution 


AIR  UNIVERSITY  LIBRARY 

ARMY  MANAGEMENT  STAFF  COLLEGE  LIBRARY 

ARMY  RESEARCH  INSTITUTE  LIBRARY 

ARMY  WAR  COLLEGE  LIBRARY 

CENTER  FOR  NAVAL  ANALYSES  LIBRARY 

DEFENSE  TECHNICAL  INFORMATION  CENTER 

HUMAN  RESOURCES  DIRECTORATE  TECHNICAL  LIBRARY 

JOINT  FORCES  STAFF  COLLEGE  LIBRARY 

MARINE  CORPS  UNIVERSITY  LIBRARIES 

NATIONAL  DEFENSE  UNIVERSITY  LIBRARY 

NAVAL  HEALTH  RESEARCH  CENTER  WILKINS  BIOMEDICAL  LIBRARY 
NAVAL  POSTGRADUATE  SCHOOL  DUDLEY  KNOX  LIBRARY 
NAVAL  RESEARCH  LABORATORY  RUTH  HOOKER  RESEARCH  LIBRARY 
NAVAL  WAR  COLLEGE  LIBRARY 

NAVY  PERSONNEL  RESEARCH,  STUDIES,  AND  TECHNOLOGY  SPISHOCK 
LIBRARY  (3) 

PENTAGON  LIBRARY 

USAF  ACADEMY  LIBRARY 

US  COAST  GUARD  ACADEMY  LIBRARY 

US  MERCHANT  MARINE  ACADEMY  BLAND  LIBRARY 

US  MILITARY  ACADEMY  AT  WEST  POINT  LIBRARY 

US  NAVAL  ACADEMY  NIMITZ  LIBRARY 


